Arrangements for digital marking and reading of items, useful in recycling

ABSTRACT

Images depicting items in a waste flow on a conveyor belt are provided to two analysis systems. The first system processes images to decode digital watermark payload data found on certain of the items (e.g., plastic containers). This payload data is used to look up corresponding attribute metadata for the items in a database, such as the type of plastic in each item, and whether the item was used as a food container or not. The second analysis system can be a spectroscopy system that determines the type of plastic in each item by its absorption characteristics. When the two systems conflict in identifying the plastic type, a sorting logic processor applies a rule set to arbitrate the conflict and determine which plastic type is most likely. The item is then sorted into one of several different bins depending on a combination of the final plastic identification, and whether the item was used as a food container or not. A variety of other features and arrangements are also detailed.

RELATED APPLICATION DATA

This application claims priority to provisional applications 63/146,631,filed Feb. 6, 2021, 63/093,207, filed Oct. 17, 2020, 63/011,195, filedApr. 16, 2020, and 63/000,471, filed Mar. 26, 2020.

The subject matter of this application expands on that of applicationSer. No. 16/435,292, filed Jun. 7, 2019 (published as 20190306385),which claims priority to provisional applications 62/854,754, filed May30, 2019, 62/845,230, filed May 8, 2019, 62/836,326, filed Apr. 19,2019, 62/830,318, filed Apr. 5, 2019, 62/818,051, filed Mar. 13, 2019,62/814,567, filed Mar. 6, 2019, and 62/812,711, filed Mar. 1, 2019.Application Ser. No. 16/435,292 is also a continuation-in-part ofapplication Ser. No. 15/823,138, filed Nov. 27, 2017 (published as20180338068), which is a continuation of application Ser. No.14/611,515, filed Feb. 2, 2015 (published as 20150302543), which claimspriority to provisional application 61/934,425, filed Jan. 31, 2014.

The subject matter of this application also expands on that ofapplication PCT/US20/22801, filed Mar. 13, 2020 (published asWO2020186234), which claims priority to applications 62/968,106, filedJan. 30, 2020, 62/967,557, filed Jan. 29, 2020, 62/956,493, filed Jan.2, 2020, and 62/923,274, filed Oct. 18, 2019.

The subject matter of this application is also related to that ofapplication Ser. No. 16/944,136, filed Jul. 30, 2020.

The disclosures of the above applications are incorporated herein byreference.

BACKGROUND AND INTRODUCTION

Applicant's patent publications US20190306385 and WO2020186234 detailnovel recycling methods and systems to help recover, by recycling orre-use, some of the millions of tons of consumer plastic that arepresently lost each year to landfills or incinerators. Disclosed inthose documents are improved ways of marking plastic items to facilitatetheir recognition, and improved methods for processing such items inmaterials recovery facilities. Various digital watermarking technologiesand improvements are particularly detailed.

The present specification builds on the teachings in those publications.The reader is presumed to be familiar with that work.

In one illustrative aspect, the technology involves a waste recoveryfacility in which items are transported for sorting on a conveyor belt.One or more cameras capture images of items on the belt. The images areprovided to two analysis systems. The first analysis system processesimagery to decode digital watermark payload data found on certain of theitems (e.g., plastic containers). This payload data is used to look upcorresponding attribute metadata for the items in a database, such asthe type of plastic in each item, and whether the item was used as afood container or not.

The second analysis system can be a spectroscopy system that determinesthe type of plastic in each item by its absorption characteristics.Sometimes the type of plastic identified by the second analysis systemconflicts with the type of plastic identified by the first analysissystem. In such case a sorting logic processor applies a rule set toarbitrate the conflict and determine which plastic type is most likely.The item is then sorted into one of several different bins depending ona combination of (a) the final plastic identification, and (b) whetherthe item was used as a food container or not.

In another embodiment the second analysis system is a convolutionalneural network trained to classify items in the imagery by theirapparent degree of contamination (e.g., external soiling or residualcontents within). Items are then sorted into different bins depending on(a) the plastic identification as determined by the first analysissystem, and (b) the contamination state (e.g., clean or dirty) asdetermined by the second analysis system.

In a variant embodiment the convolutional neural network is trained todistinguish plastic bottles with caps from plastic bottles without caps.Again, items are sorted into different bins based on data from both ofthe analysis systems, with capped bottles of a first plastic type beingsorted into one bin, and uncapped plastic bottles of that first plastictype being sorted into a different bin.

The foregoing and a great number of other features and aspects of thepresent technology will be more readily apparent from the followingdetailed description, which proceeds with reference to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system that can employ certain aspects of thepresent technology.

FIG. 2A show an illustrative watermark reference signal in the pixeldomain, and FIG. 2B shows the same signal expressed in the Fouriermagnitude domain.

FIG. 3 depicts a partially-assembled illumination module.

FIG. 4 depicts a geometrical relationship between light sources, acamera, and an item being imaged.

FIG. 5 shows relative sizes of different features within a lattice ofcells scaled for 75 watermark elements per inch.

FIG. 6 illustrates use of logos as marks in a sparse watermark pattern.

FIG. 7 schematically illustrates a breakbeam arrangement for sensingempty excerpts of a conveyor belt.

FIGS. 8 and 8A schematically illustrate a laser line-based arrangementfor sensing empty excerpts of a conveyor belt.

FIG. 9 illustrates how newly-captured belt imagery can be correlatedagainst previously-captured belt imagery to identify an empty region ofbelt.

FIG. 10 illustrates a pattern of markings that can be formed on aconveyor belt to facilitate detection of empty excerpts of the belt.

FIG. 11 shows a plastic lid thermoformed with various signal patternpatches.

FIGS. 12A and 12B shows a variety of signal patterns, with associatedparameters.

FIG. 13 details certain robustness measurements made on thermoformedsignal patterns of different varieties.

FIG. 14 shows an excerpt of one thermoformed signal pattern.

FIG. 14A is a variant of FIG. 14 .

FIG. 15 shows an excerpt of a different thermoformed signal pattern.

FIG. 15A is a variant of FIG. 15 .

FIG. 16 shows a laser-marked plastic bottle.

FIG. 16A is an excerpt taken from FIG. 16 .

FIG. 17 illustrates a system employing employ certain aspects of thepresent technology.

FIGS. 18A-18D illustrate how a bottle bearing a machine-readable markthat identifies the bottle and its shape, enables a range of possiblebottle positions to be determined.

FIG. 19 shows a flowchart of an exemplary embodiment employing an aspectof the present technology.

FIG. 20 shows an annotated map of an image frame produced by a trainedclassifier.

FIG. 21 illustrates a system employing employ certain aspects of thepresent technology.

FIG. 22 shows the profile of an exemplary bottle that may be labeledwith a shrink-fit plastic sleeve.

FIG. 23 shows how a waist of the FIG. 22 bottle profile changes theaspect ratio of watermark blocks, when a uniform array of blocks isemployed (on the left), and when a vertically pre-warped array of blocksis employed (on the right).

FIGS. 24A and 24B show two alternative ways of marking a sector of anannulus with an array of signal tiles.

DETAILED DESCRIPTION

There is a critical need for high-reliability identification of plasticitems, e.g., for sorting waste streams. Digital watermarks are suited tothis task.

Digital watermarks provide 2D optical code signals that enable machinevision in waste sorting systems to determine the types of material(e.g., variety of plastic) in each object. Encoded identificationsignals imparted into and onto containers (e.g., via printed labels,textured molds, laser engraving of plastic, etc.) can be sensed and usedto control sorting based on container material and other factors. Sincedigital watermark signals can be spread over a container and/or itslabels in ways that provide identification even when the object isdamaged, soiled or partially occluded, the technology is particularlyadvantageous for waste sorting purposes.

An illustrative recycling apparatus employing aspects of the presenttechnology is shown in FIG. 1 and employs one or more cameras, andtypically light sources, to capture imagery depicting watermarkedplastic items traveling in a waste stream on a conveyor belt. Dependingon implementation, the conveyor area imaged by a camera system (i.e.,its field of view) may be as small as about 2 by 3 inches, or as largeas about 20 by 30 inches, or larger—primarily dependent on camera sensorresolution and lens focal length. In some implementations, multipleimaging systems are employed to capture imagery that collectively spanthe width of the conveyor. A conveyor may be up to two meters in widthin a mass-feed system. (Singulated-feed systems, in which items aremetered onto the conveyor one at a time, are narrower, e.g., 50 cm inwidth.) Conveyor speeds of 1-5 meters/second are common.

Image frames depicting an item are provided to a detector that decodeswatermark payload data for an item from small blocks of imagery. Thewatermark payload data comprises a short identifier (e.g., 50-100 bits),which is associated with a collection of related metadata in a database(sometimes termed a “resolver database”). This metadata can detail alengthy set of attributes about the plastic used in the item, such asits chemistry and properties, e.g., its melt index, melt flow ratio,resin specific gravity, bulk density, melt temperature, fillers andadditives, color pigments, etc. The metadata can further providenon-plastic information, such as dimensions of the item, whether theitem was used as a food container or not, whether the package is amulti-layer composite or includes a sleeve, the corporate brandresponsible for the item, etc.

The locations of decoded watermark signal blocks within captured imageframes are mapped to corresponding physical areas on the conveyor belt.The belt speed is known, so the system can predict whenwatermark-identified items will be in position to be diverted from thebelt into an appropriate receptacle, or onto a selected furtherconveyor. Familiar diversion means can be used, such as force air“blowout.”

Plastic items can be encoded with multiple watermarks. One watermark canbe printed—typically by ink—on a label or sleeve applied to the item (orprinted on the item itself), and another can be formed by 3D texturingof the plastic surface. The payload of a printed watermark commonlyconveys a retail payload (e.g., a GTIN, a Global Trade Item Number),which is designed primarily for use by a point-of-sale terminal scanner,as it contains or points to a product name, price, weight, expirationdate, package date, etc., to identify and price an item at a retailcheckout. (“Points to” refers to use of the payload information toidentify a corresponding database record, from which further informationabout the item is obtained.) The texture watermark may comprise the samepayload, or one specific to recycling, e.g., containing or pointing todata relating to the plastic.

Watermarking Technology

Certain inventive aspects of the present technology concern improvementsto watermarking technology, so we provide an introductory discussion ofillustrative watermark encoding and decoding arrangements. (Thefollowing details are phrased in the context of print, but theapplication of such methods to surface texturing is straightforward,e.g., given teachings elsewhere in this disclosure and in the citeddocuments.)

In an exemplary encoding method, a plural-symbol message payload (e.g.,47 binary bits, which may represent a product's Global TradeIdentification Number (GTIN), or a container identification code,together with 24 associated CRC bits), is applied to an error correctioncoder. This coder transforms the symbols of the message payload into amuch longer array of encoded message elements (e.g., binary or M-aryelements) using an error correction method. (Suitable coding methodsinclude block codes, BCH, Reed Solomon, convolutional codes, turbocodes, etc.) The coder output may comprise hundreds or thousands ofbinary bits, e.g., 1024, which may be termed raw signature bits. Thesebits may be scrambled by XORing with a scrambling key of the samelength, yielding a scrambled signature.

Each bit of the scrambled signature modulates a pseudorandom noisemodulation sequence (spreading carrier) of length 16, e.g., by XORing.Each scrambled signature bit thus yields a modulated carrier sequence of16 “chips,” producing an enlarged scrambled payload sequence of 16,384elements. This sequence is mapped to elements of a square block having128×128 embedding locations in accordance with data in a map or scattertable, yielding a 2D payload signature pattern comprised of 128×128watermark elements (“waxels”). In a particular embodiment, the scattertable assigns 4 chips for each scrambled signature bit to each of four64×64 quadrants in the 128×128 block.

Each location in the 128×128 block is associated with a waxel (chip)value of either 0 or 1 (or −1 or 1, or black or white)—with about halfof the locations having each state. This bimodal signal is frequentlymapped to a larger bimodal signal centered at an eight-bit greyscalevalue of 128, e.g., with values of 95 and 161. Each of these embeddinglocations may correspond to a single pixel, resulting in a 128×128 pixelwatermark message block. Alternatively, each embedding location maycorrespond to a small region of pixels, such as a 2×2 patch, termed a“bump,” resulting in a 256×256 pixel message block.

A synchronization component is commonly included in a digital watermarkto help discern parameters of any affine transform to which thewatermark has been subjected prior to decoding, so that the payload canbe correctly decoded. A particular synchronization component takes theform of a reference signal comprised of a dozen or more 2D sinusoids ofdifferent frequencies and pseudorandom phases in the pixel (spatial)domain, which corresponds to a pattern or constellation of peaks ofpseudorandom phase in the Fourier (spatial frequency) domain. Suchalternate representations of an illustrative reference signal are shownin FIG. 2A (pixel domain) and FIG. 2B (Fourier domain). As a matter ofpractice, this signal is commonly defined in the Fourier domain andtransformed into the pixel domain at a size corresponding to that of thewatermark message block, e.g., 256×256 pixels. This pixel referencesignal, which may comprise floating-point values between −1 and 1, canbe magnitude-scaled to a range of −40 to 40. Such reference signalelements are then combined with corresponding elements of the 256×256pixel payload block to yield a final watermark signal block, e.g.,having values ranging from 55 (i.e., 95-40) to 201 (i.e., 161+40). Forprint applications such signal can then be summed with host imagery,after first scaling-down in magnitude to render it inconspicuous.

If such a watermark signal block is rendered at a spatial resolution of300 dots per inch (DPI), a signal block of about 0.85 inches squareresults. Since the 0.85 inch side dimension corresponds to 128 waxels,this works out to 150 waxels per inch. (Naturally, other sizes can beemployed, e.g., 75, 200, 300 and 750 waxels per inch, etc.) Such blockscan be tiled edge-to-edge for marking a larger surface—in some casesspanning an object completely.

The just-described watermark signal may be termed a “continuous tone”watermark signal. In print it is often characterized by multi-valueddata, i.e., not being just on/off (or 1/0, or black/white)—thus the“continuous” moniker. Each pixel of the host content (or region withinthe host content) is associated with one corresponding element of thewatermark signal. A majority of pixels in a host image (or image region)are changed in value by combination with their corresponding watermarkelements. The changes are typically both positive and negative, e.g.,changing the local luminance of the imagery up in one location, whilechanging it down in another. And the changes may be different indegree—some pixels are changed a relatively smaller amount, while otherpixels are changed a relatively larger amount. Typically, the amplitudeof the watermark signal is low enough that its presence within the imageescapes notice by casual viewers (i.e., it is steganographic).

(Due to the highly redundant nature of the encoding, some embodimentscan disregard pixel changes in one direction or another. For example,one such embodiment only changes pixel values in a positive direction.Pixels that would normally be changed in a negative direction are leftunchanged. The same approach can be used with surface texturing, i.e.,changes can be made in one direction only.)

In a variant continuous tone print watermark, the signal acts not tochange the local luminance of artwork pixels, but rather their color.Such a watermark is termed a “chrominance” watermark (instead of a“luminance” watermark). An example is detailed, e.g., in U.S. Pat. No.9,245,308.

“Sparse” or “binary” watermarks are different from continuous tonewatermarks. They do not change a majority of pixel values in the hostimage (or image region). Rather, they have a print density (which maysometimes be set by the user) that typically results in marking betweenabout 3% and 45% of pixel locations in the image. Adjustments areusually all made in the same direction, e.g., reducing luminance. Sparseelements are commonly bitonal, e.g., being either white or black.Although sparse watermarks may be formed on top of other imagery, theyare often presented in regions of artwork that are blank or colored witha uniform tone. In such cases a sparse marking may contrast with itsbackground, rendering the marking visible to casual viewers. Althoughsparse marks can take the form of a field of seemingly-random dots, theycan also take the form of line structures, as detailed elsewhere. Aswith continuous tone watermarks, sparse watermarks generally take theform of signal blocks that are tiled across an area of imagery.

A sparse watermark can be produced from a continuous-tone watermark invarious ways. One is by thresholding. That is, the darkest elements ofthe summed reference signal/payload signal blocks are copied into anoutput signal block until a desired density of dots is achieved. Such awatermark may be termed a thresholded binary watermark.

Patent publication US20170024840 details various other forms of sparsewatermarks. In one embodiment, a watermark signal generator starts withtwo 128×128 inputs. One is a payload signal block, with its locationsfilled with a binary (0/1, black/white) enlarged scrambled payloadsequence, as described above. The other is a spatial domain referencesignal block, with each location assigned a floating point numberbetween −1 and 1. The darkest (most negative) “x”% of these referencesignal locations are identified, and set to black; the others are set towhite. Spatially-corresponding elements of the two blocks are ANDedtogether to find coincidences of black elements between the two blocks.These elements are set to black in an output block; the other elementsare left white. By setting “x” higher or lower, the output signal blockcan be made darker or lighter. Such a code may be termed an ANDed, or aType 1, binary watermark.

Publication US20190332840 details additional sparse encodingembodiments. One embodiment uses a reference signal generated at arelatively higher resolution (e.g., 384×384 pixels), and a payloadsignature spanning a relatively lower resolution array (e.g., 128×128).The latter signal has just two values (i.e., it is bitonal); the formersignal has more values (i.e., it is multi-level, such as binarygreyscale or comprised of floating-point values). The payload signal isinterpolated to the higher resolution of the reference signal, and inthe process is converted from bitonal form to multi-level. The twosignals are combined at the higher resolution (e.g., by summing in aweighted ratio), and a thresholding operation is applied to the resultto identify locations of extreme (e.g., dark) values. These locationsare marked to produce a sparse block (e.g., of 384×384). The thresholdlevel establishes the dot density of the resulting sparse mark. Such acode may be termed an interpolated, or a Type 2, binary watermark.

A different embodiment orders samples in a block of a reference signalby value (darkness), yielding a ranked list of the darkest N locations(e.g., 1600 locations), each with a location (e.g., within a 128×128element array). The darkest of these N locations may be always-marked inan output block (e.g., 400 locations, or P locations), to ensure thereference signal is strongly expressed. The others of the N locations(i.e., N-P, or Q locations) are marked, or not, depending on values ofmessage signal data that are mapped to such locations (e.g., by ascatter table in the encoder). Locations in the sparse block that arenot among the N darkest locations (i.e., neither among the P or Qlocations) never convey watermark signal, and they are consequentlyaffirmatively ignored by the decoder. By setting the number N larger orsmaller, sparse marks with more or fewer dots are produced. Thisembodiment is termed the “fourth embodiment” in earlier-citedpublication US20190332840, and may also be termed a Type 3 binarywatermark.

In generating a binary (sparse) mark, a spacing constraint can beapplied to candidate mark locations to prevent clumping. The spacingconstraint may take the form of a keep-out zone that is circular,elliptical, or of other (e.g., irregular) shape. The keep-out zone mayhave two, or more, or less, axes of symmetry (or none). Enforcement ofthe spacing constraint can employ an associated data structure havingone element for each location in the tile. As dark marks are added tothe output block, corresponding data is stored in the data structureidentifying locations that—due to the spacing constraint—are no longeravailable for possible marking.

This process can involve assessing candidate dot locations to assurethat a keep-out region around previously-selected dots is observed.Consider a candidate dot at row 11 and column 223. If the keep-outregion is a distance of 4 elements (pixels), then one implementationlooks to see if any dot has previously been selected in rows 7-15. Thisset of previously-selected dots is further examined to determine whetherany is found in columns 219-227. If so, then the distance between suchpreviously-selected dot and the candidate dot is determined, bycomputing the square root of the sum of the row difference, squared, andthe column distance, squared. If any such distance is less than 4, thenthe candidate dot is discarded.

Applicant has found that a substantial computational saving can beachieved by a different algorithm. This different algorithm maintains alook-up data structure (table) having dimensions equal to that of thedense code, plus a border equal to the keep-out distance. The datastructure thus has dimensions of 392 rows×392 columns, in the case of a384×384 element dense code (and a keep out of 4). Each element in thedata structure is initialized to a value of 0—signifying that thecorresponding position in the sparse block is available for a dot.

When a first dot is selected for inclusion in the sparse block, thecorresponding location in the data structure—and neighboring locationswithin a distance of 4—are changed to a value of 1—signifying that theyare no longer available for a dot. When a second candidate dot isconsidered at a given row/column of the sparse signal block, thecorresponding row/column of the data structure is checked to see if itsvalue is 0 or 1. If 0, the dot is written to the sparse output block,and a corresponding neighborhood of locations in the data structure arechanged in value to 1—preventing any other dots from violating the keepout constraint. This process continues, with the location of eachcandidate dot being checked against the corresponding location in thedata structure and, if still a 0, the dot is written to the sparseoutput block and the data structure is updated accordingly.

In some embodiments, the reference signal can be tailored to have anon-random appearance (in contrast to that of FIG. 2A), by varying therelative amplitudes of spatial frequency peaks, so that they are not allof equal amplitude. Such variation of the reference signal hasconsequent effects on the sparse signal appearance.

A sparse pattern can be rendered in various forms. Most straight-forwardis as a seemingly-random pattern of dots. But more artistic renderingsare possible, including Voronoi and Delaunay line patterns, and stipplepatterns, as detailed in our patent publication US20190378235.

Other overt, artistic patterns conveying watermark data are detailed inpatent publication US20190139176. In one approach, a designer creates acandidate artwork design or selects one from a library of designs.Vector art in the form of lines or small, discrete print structures ofdesired shape work well in this approach. A payload is input to a signalgenerator, which generates a raw data signal in the form oftwo-dimensional tile of data signal elements. The method then edits theartwork at spatial locations according to the data signal elements atthose locations. When artwork with desired aesthetic quality androbustness is produced, it is applied to an object.

Other techniques for generating visible artwork bearing a robust datasignal are detailed in assignee's patent publications US20190213705 andUS20200311505. In some embodiments, a neural network is applied toimagery including a machine-readable code, to transform its appearancewhile maintaining its machine readability. One particular method trainsa neural network with a style image having various features. (Van Gogh'sThe Starry Night painting is often used as an exemplary style image.)The trained network is then applied to an input pattern that encodes aplural-symbol payload. The network adapts features from the style image(e.g., distinctive colors and shapes) to express details of the inputpattern, to thereby produce an output image in which features from thestyle image contribute to encoding of the plural-symbol payload. Thisoutput image can then be used as a graphical component in productpackaging, such as a background, border, or pattern fill. In someembodiments, the input pattern is a watermark pattern, while in othersit is a host image that has been previously watermarked.

Still other such techniques do not require a neural network. Instead, acontinuous tone watermark signal block is divided into sub-blocks. Astyle image is then analyzed to find sub-blocks having the highestcorrelation to each of the watermark signal sub-blocks. Sub-blocks fromthe style image are then mosaiced together to produce an output imagethat is visually evocative of the style image, but has signalcharacteristics mimicking the watermark signal block. Yet anothertechnique starts with a continuous tone watermark, divides it intosub-blocks, and combines each sub-block with itself in various states ofrotation, mirroring and/or flipping. This yields a watermark blockcomprised of stylized sub-blocks that appear somewhat likegeometrically-patterned symmetrical floor tiles.

Watermark reading has two parts: finding a watermark, and decoding thewatermark.

In one implementation, finding the watermark (sometimes termed watermarkdetection) involves analyzing a received frame of captured imagery tolocate the known reference signal, and more particularly to determineits scale, rotation, and translation.

The received imagery is desirably high-pass filtered so that the finedetail of the watermark code is maintained, while the low frequencydetail of the item on which it is marked is relatively attenuated.Oct-axis filtering can be used.

In one oct-axis filtering arrangement, each pixel is assigned a newvalue based on some function of the original pixel's value relative toits neighbors. An exemplary embodiment considers the values of eightneighbors—the pixels to the north, northeast, east, southeast, south,southwest, west and northwest. An exemplary function sums a −1 for eachneighboring pixel with a lower value, and a +1 for each neighboringpixel with a higher value, and assigns the resulting value to thecentral pixel. Each pixel is thus re-assigned a value between −8 and +8.(These values may all be incremented by 8 to yield non-negative values,with the results divided by two, to yield output pixel values in therange of 0-8.) Alternatively, in some embodiments only the signs ofthese values are considered—yielding a value of −1, 0 or 1 for everypixel location. This form can be further modified to yield a two-stateoutput by assigning the “0” state, either randomly or alternately, toeither “−1” or “1.” Such technology is detailed in Digimarc's U.S. Pat.Nos. 6,580,809, 6,724,914, 6,631,198, 6,483,927, 7,688,996, 8,687,839,9,544,516 and 10,515,429. (A variant filtering function, the “freckle”transform, is detailed in U.S. Pat. No. 9,858,681. A further variant,“oct-vector,” is detailed in pending application Ser. No. 16/994,251,filed Aug. 14, 2020.)

A few to a few hundred candidate blocks of filtered pixel imagery(commonly overlapping) are selected from the filtered image frame in anattempt to identify one or more watermarked items depicted in the imageframe. (An illustrative embodiment selects 300 overlapping blocks.) Eachselected block can have dimensions of the originally-encoded watermarkblock, e.g., 64×64, 128×128, 256×256, etc. We focus on the processingapplied to a single candidate block, which is assumed to be 128×128pixels in size.

To locate the reference signal, the selected pixel block is firsttransformed into the Fourier domain, e.g., by a Fast Fourier Transform(FFT) operation. If a watermark is present in the selected block, thereference signal will be manifested as a constellation of peaks in theresulting Fourier magnitude domain signal. The scale of the watermark isindicated by the difference in scale between the original referencesignal constellation of peaks (FIG. 2B), and the constellation of peaksrevealed by the FFT operation on the received, filtered imagery.Similarly, the rotation of the watermark is indicated by the angularrotation difference between the original reference signal constellationof peaks (FIG. 2B), and the constellation of peaks reveals on the FFToperation on the received, filtered imagery.

A direct least squares, or DLS technique is commonly used to determinethese scale and rotation parameters, with each of a thousand or morecandidate, or “seed,” affine transformations of the known referencesignal being compared to the magnitude data from the FFT transform ofthe input imagery. The parameters of the one or more seed affinetransforms yielding FFT magnitude data that most nearly matches that ofthe block of filtered input imagery are iteratively adjusted to improvethe match, until a final scale/rotation estimate is reached thatdescribes the pose of the reference signal within the analyzed block ofimagery.

Once the scale and rotation of the watermark within the received imageblock are known, the watermark's (x,y) origin (or translation) isdetermined. Methods for doing so are detailed in our U.S. Pat. Nos.6,590,996, 9,959,587 and 10,242,434 and can involve, e.g., a FourierMellin transform, or phase deviation methods. (The just-noted patentsalso provide additional detail regarding the DLS operations to determinescale and rotation; they detail decoding methods as well.)

Once known, the scale, rotation and translation information(collectively, “pose” information) establishes a spatial relationshipbetween waxel locations in the 128×128 watermark signal block, andcorresponding locations within the image signal block. That is, one ofthe two signal blocks can be scaled, rotated and shifted so that eachwaxel location in the watermark code is spatially-aligned with acorresponding location in the image block.

Next, the original image data is geometrically transformed in accordancewith the just-determined pose information and is resampled to determineimage signal values at an array of 128×128 locations corresponding tothe locations of the 128×128 waxels. Since each waxel location typicallyfalls between four pixel locations sampled by the camera sensor, it isusually necessary to apply bilinear interpolation to obtain an estimateof the image signal at the desired location, based on the values of thenearest four image pixels. The known reference signal has served itspurposes at this point, and now just acts as noise, so it can besubtracted if desired. Oct-axis filtering is again applied. This yieldsa 128×128 waxel-registered array of filtered image data. From this datathe watermark payload can be decoded.

In particular, the decoder examines the mapped locations for each of the16 chips corresponding to a particular bit of the scrambled signature,and inverts each filtered image value—or not—in accordance with acorresponding element of the earlier-applied XOR spreading carrier. Theresulting 16 values are then summed—optionally after each is weighted bya linear pattern strength metric (or grid strength metric) indicatingstrength of the reference signal in the watermark sub-block from whichthe value was sampled. (Suitable strength metrics are detailed in U.S.Pat. Nos. 10,217,182 and 10,506,128.) The sign of this sum is anestimate of the scrambled signature bit value—a negative value indicates−1, a positive value indicates +1. The magnitude of the sum indicatesreliability of the estimated bit value. This process is repeated foreach of the 1024 elements of the scrambled signature, yielding a 1024element string. This string is descrambled, using the earlier-appliedscrambling key, yielding a 1024 element signature string. This string,and the per-bit reliability data, are provided to a Viterbi softdecoder, which returns the originally-encoded payload data and CRC bits.The decoder then computes a CRC on the returned payload and compares itwith the returned CRC. If no error is detected, the read operationterminates by outputting the decoded payload data, together withcoordinates—in the image frame of reference—at which the decoded blockis located (e.g., its center, or its upper right corner “origin”). Thepayload data is passed to the database to acquire corresponding itemattribute metadata. The coordinate data and metadata needed for sortingare passed to a sorting logic (diverter) controller. Metadata not neededfor sorting but logged for statistical purposes are passed to a logfile.

In some embodiments, pose parameters are separately refined foroverlapping sub-blocks within the 128×128 waxel block. Each waxel mayfall into, e.g., four overlapping sub-blocks, in which case there may befour interpolated, filtered values for each waxel, each corresponding toa different set of pose parameters. In such case these four values canbe combined (again, each weighted in accordance with a respective gridstrength metric), prior to inversion—or not—in accordance with thecorresponding element of the earlier-applied XOR spreading carrier.

Relatedly, once pose parameters for the image block are known,surrounding pixel data can be examined to see if the reference signal ispresent there too, with the same or similar pose parameters. If so, caseaddition chip information can be gathered. (Since the watermark block istypically tiled, chip values should repeat at offsets of 128 waxels invertical and horizontal directions.) Chip values from such neighboringlocations can be weighted in accordance with the grid strength of thesub-block(s) in which they are located, and summed with other estimatesof the chip value, to gain still further confidence.

The just-described accumulation of chip data from beyond a singlewatermark block may be termed intraframe signature combination.Additionally, or alternatively, accumulation of chip or waxel data fromthe same or corresponding locations across patches depicted in differentimage frames can also be used, which may be termed interframe signaturecombination.

In some embodiments plural frames that are captured by the camerasystem, e.g., under different illumination conditions and/or fromdifferent viewpoints, are registered and combined before submission tothe detector system.

In print, the different values of watermark elements are signaled by inkthat causes the luminance (or chrominance) of the substrate to vary. Intexture, the different values of watermark elements are signaled byvariations in surface configuration that cause the reflectance of thesubstrate to vary. The change in surface shape can be, e.g., a bump, adepression, or a roughening of the surface.

Such changes in surface configuration can be achieved in various ways.For mass-produced items, molding (e.g., thermoforming, injectionmolding, blow molding) can be used. The mold surface can be shaped by,e.g., CNC or laser milling, or chemical or laser etching. Non-moldapproaches can also be used, such as forming patterns on the surface ofa container by direct laser marking.

Laser marking of containers and container molds is particularlypromising due to the fine level of detail that can be achieved.Additionally, laser marking is well-suited for item serialization—inwhich each instance of an item is encoded differently.

One application of serialization is to identify reusable bottles thatare submitted for refilling, e.g., by a drink producer. After a bottlehas been refilled, e.g., 20 times, it can be retired from service. See,e.g., patent publication US20180345326.

More generally, watermark serialization data can be used to help trackindividual bottles and other items of packaging through their respectivelifecycles, from fabrication to recycling/re-use, and to provide datathat makes possible an incentive system—including refunds of fees andrebates of taxes—to help encourage involvement by the many differentparticipants needed to achieve the vision of a circular economy (e.g.,bottle producers, brands, distributors, retailers, consumers, wastecollection companies, material recovery facilities, recyclers, extendedproducer responsibility organizations, etc.).

In addition to the references cited elsewhere, details concerningwatermark encoding and reading that can be included in implementationsof the present technology are disclosed in applicant's previous patentfilings, including U.S. Pat. Nos. 6,985,600, 7,403,633, 8,224,018,10,958,807, and in pending application Ser. No. 16/823,135, filed Mar.18, 2020.

Further information about thermoforming (molding) of plastic items isdetailed in application 63/076,917, filed Sep. 10, 2020. Furtherinformation about injection molding is detailed in application63/154,394, filed Feb. 26, 2021. Further information about laser markingof containers (which technology is also applicable to laser marking ofmolds) is detailed in application 63/113,700, filed Nov. 13, 2020.

Illustrative Hardware

The following discussion provides a summary of an illustrative imagingsystem, including illumination and imaging components.

An exemplary illumination system for watermark image capture isfashioned from circuit board modules. A partially-assembled example isshown in FIG. 3 . This board is populated with LEDs of the Cree XP-E2series, arrayed as 25 triples, each with its own lens (e.g., CarcloTechnical Plastic part number 10510). Additional information about thisboard, and several variants, are detailed in patent publicationWO2020186234.

Such modules can be placed edge-to-edge to span the width of theconveyor belt. As shown in FIG. 1 , the belt is desirably illuminatedfrom two directions.

The LEDs may all be of the same color, or LEDs of different colors canbe included. In an exemplary arrangement, blue, red and infrared LEDsare employed, each with a spectral peak bandwidth (FWHM) of 40 or 30nanometers or less at respective wavelengths of 450, 660 and 730nanometers. These LEDs can be operated in tandem, but more commonly areoperated in a monochrome fashion, e.g., a flash of blue, followed by aflash of red, followed by a flash of infrared. Each flash issynchronized to capture of a frame by the camera system. In sucharrangement each frame in a triplet of frames is captured under adifferent illumination spectrum. (Naturally other colors can beemployed, including white, green and ultraviolet.)

A variant illumination module does not use the circular lens assembliesof FIG. 3 , but rather uses linear lenses. This permits the LEDs, andthe rows of LEDs, to be spaced more closely, thereby providing morelight for a given size module. Suitable linear lenses are available fromKhatod (e.g., the PL1629NAST), Fusion Optix (e.g., the LEDMateLinear-Convex), Carclo (e.g., model 10398) and Gaggione (e.g., LLL15N7).Desirably, each lens projects a beam that spans the camera field of viewalong the length dimension of the belt (e.g., 10-20 cm, nominally 14cm), when spaced 50 cm from the belt. The LEDs that are mounted in a rowunder a common lens may all be of the same color, or each row mayinclude multiple colors. The LEDs may be spaced as closely within therow as thermal considerations permit.

In another embodiment, one or more elliptical light shaping diffusorsheets are employed. These sheets scatter LED or laser illumination,incident on one side, to produce a shaped pattern exiting the otherside. Different output patterns are available, such as with a spread ofbetween 1 to 60 degrees in one dimension, and a spread of between 10 and80 degrees in the perpendicular dimension. The longer dimension (whichin a particular embodiment may be 40-60 degrees) is typically orientedto illuminate across the width dimension of the belt.

By using such a diffusor over circuit board modules of LEDs, the LEDsmay be spaced still more densely because the separate lens assembliesmay be omitted. (Exemplary LEDs are less than 4 mils on a side,permitting up to 25 to be mounted in a 2×2 cm area.) Denser placementallows brighter illumination, and enables use of a greater diversity ofLED colors than is described above. Still brighter illumination may beachieved by selection of narrower dispersion patterns. For example, a45×8 degree dispersion pattern generally provides doubles the lightintensity of a 45×16 degree dispersion pattern—all other things beingequal. Increased illumination permits shorter exposure intervals and/orsmaller lens apertures, leading to reduction of motion blur and/orincrease in depth of field. (Moreover, such diffusors typically haveefficiencies of over 90%, as contrasted with efficiencies of below 90%for plastic LED lenses.)

Luminit LLC, of Torrance, Calif., and Bright View TechnologiesCorporation, of Durham, N.C., are suppliers of suitable diffusor sheets.

Typically each light source has an apparent width of at least 5 cm. Thelight sources are pulsed at the camera frame rate and desirably areactive only when the camera is exposing an image.

Applicant surprisingly has found that watermark detection from crumpledobjects (e.g., plastic bottles) sometimes works best if the imagery isanalyzed in elongated excerpts, rather than square. For example, insteadof operating on patches of imagery sized to span about 128×128 or 32×32waxels, better results may be achieved by operating on imagerycorresponding to 32×16, or 32×8, waxels. In such case, the longerdimension of the analysis excerpt is desirably aligned to be parallel toany elongation in the illumination pattern. For example, if theillumination is shaped to span a greater distance along the width of thebelt than along its length, then the analysis excerpts of imagery aredesirably taken with their longer axes oriented in the pixel directionthat corresponds to the width of the belt.

Applicant's U.S. patent application 63/117,828, filed Nov. 24, 2020,provides additional details on suitable illumination systems.

The illumination system is desirably positioned as close as the belt aspossible, to provide the brightest illumination and thereby permit theshortest possible camera capture (exposure) intervals. However,sufficient clearance must be provided to enable items to pass beneath onthe belt. In a particular embodiment, a distance of between 15-20 cm isused. Depending on the types of items on the belt, a higher clearance(e.g., of 25-60 cm.) may be required. The sorting system may include acrusher that serves to reduce height variation of the plastic surfacesbefore items are imaged. (Crushing also reduces tumbling.)

Specular reflection from smooth plastic surfaces can be a hindrance.Sometimes, however, it can be a help—depending on circumstance. (Imagingblack plastic is one circumstance in which it can be a help. Another iswhere marking effects a roughening of a plastic surface, so thatmarkings are distinguished in captured imagery by localized absences ofspecular reflection.) One advantageous arrangement employs pluralseparately-operable light sources that are positioned—relative to thecamera—in manners configured so that one (or more) is adapted to lead tospecular reflection in captured imagery, while one (or more) is adaptedto avoid specular reflection in captured imagery.

Turning to the camera, the larger the sensor, the more sensitive it is,and the shorter the exposures can be. Desirably the sensor has pixelslarger than 3.5 micrometers on a side, and preferably larger than 5micrometers on a side. Ideally, sensors with pixels of 10 or 15micrometer size would be used, although costs are a factor. (An exampleis the SOPHIA 2048B-152 from Princeton Instruments—a 2K×2K sensor, witha pixel size of 15 micrometers.) An alternative is to use “binning” witha higher resolution sensor, e.g., a 2.5K×2.5K sensor with 5 micrometerpixels, in which adjoining 2×2 sets of pixels are binned together toyield performance akin to that of a 1.25K×1.25K sensor with 10micrometer pixels. Suitable candidates include the Sony IMX420 sensor(with 9 micrometer effective pixel size after binning, and with a 10-bitanalog-to-digital converter) and the Sony IMX425 sensor (again with 9micrometer effective pixel size, but with a 12-bit ADC). Global shutterimage capture is desirably used (as contrasted with rolling shutter) toavoid motion artifacts.

Either monochrome or color camera sensors can be used. Some printedlabels are encoded using “chroma” watermarking in which, e.g., cyan andmagenta inks are used in combination. These two inks have differentspectral reflectance curves which, when illuminated by white(red-green-blue) illumination, enable differences between red- and blue-(and/or green-) channel camera responses to be subtracted to yield animage in which the watermark signal is accentuated. (See, e.g., U.S.Pat. No. 9,245,308.) Yet despite the signal increase achieved by suchtechnique, applicant has found that illuminating such labels with redlight alone, and sensing with a monochrome sensor, can yield strongerand less noisy recovered watermark signals. (Moreover, red LEDs are moreefficient than, e.g., green and blue LEDs—sometimes by a factor of twoor more. This translates to less heat, which in turn allows more LEDs tobe used, producing greater luminous flux output.)

In still other embodiments, printed labels can be encoded with machinereadable data (e.g., sparse watermark patterns) formed with yellow ink,for encoding of recycling-related data.

Sensitivity of human vision is particularly acute in the green spectrum,so watermark data is often not encoded in a green color channel ofproduct artwork, in order to help keep the marking imperceptible. Thus,in some embodiments, illumination and camera systems that minimize useof green (e.g., but instead emphasize blue and higher wavelengths upinto ultraviolet, and red and lower wavelengths down into infrared) areused. One sensor optimized for digital watermark reading—in non-greenvisible wavelengths—is detailed in our U.S. Pat. No. 10,455,112. Aparticular embodiment detailed in that patent uses a color filter arrayover a monochrome sensor, in which there are three magenta-filteredphotocells for every green-filtered photocell.

The lens used with the camera should minimize barrel distortion andchromatic aberration (e.g., with consistent focus at both blue andinfrared, such as at 450 and 730 nanometers). Lenses in the FujinonCF-ZA-1S series are satisfactory. The lens should be focused at half ofthe camera depth of field, e.g., 5 cm from the surface of the belt. 50and 35 mm lenses have been used successfully, with longer lenses usuallybeing preferred to lessen perspective distortion.

In an exemplary system the belt moves at about 3-5 meters per second.The camera system desirably looks straight down at the belt (i.e., withthe lens axis perpendicular to the belt) and captures monochrome framesat a rate of 150-700 frames per second, and most typically at a rate of300-500 frames per second. Exposure times are normally 100 microsecondsor less, with 33 to 66 microseconds being more usual. An HB-1800-S-Mcamera system by Emergent Vision Technologies is suited for such capturerequirements and employs the earlier-referenced Sony IMX425 sensor. (Ifdesired, multiple cameras with lower frame capture rates and overlappingfields of view can be synchronized together to meet the frame raterequirements.) The camera system depth of field is typically at least 5cm, with 10-15 cm or more being preferred. Desirably the lens apertureis f/5.6 or smaller, such as f/8 or f/11. The distance between thecamera and the belt is again limited by the constraint of needingadequate clearance for items to pass underneath.

The camera optics are desirably chosen, in conjunction with the imagingdistance, so that captured imagery depicts items in the middle of thedepth of field with a resolution of about 0.7 to 2 pixels per waxel, andmore usually between 1 and 1.5 pixels per waxel. (If captured imageframes are 1280×1024 pixels, and the 1024 pixels depicts a length ofbelt measuring 14 cm, this works out to a sampling resolution of 73pixels per cm, or about 185 pixels per inch. At 150 waxels per inch,this is 1.23 pixels per waxel.)

If the belt is moving at 5 meters per second, and the camera system isproviding 500 frames per second of imagery, then the computationalresources needed to process the imagery from a camera may be met using,e.g., between 4 and 7 Intel i9 9960X 16-core AVX512 CPUs. In aparticular embodiment, the imagery from a camera is provided to anexecution thread on one core which serves as a dispatcher process,distributing the imagery to other cores and threads based on theircurrent utilization.

The field of view of a single camera may be about 18×14 centimeters. Thewidth of the entire belt is typically imaged by providing multiplecameras, with fields of view of adjacent cameras overlapping by 2 cm orso.

The illumination system can be pulsed and synchronized with the camerasystem and can be cycled through different light configurations, suchas: (a) capturing alternate image frames with infrared, then blue; (b)capturing alternate image frames with the first frame illuminated withinfrared plus blue, and the next frame illuminated with red; and (c)capturing sequences of three frames: red, infrared, blue. Each image canbe tagged with metadata indicating the color illumination with which itwas captured.

The spatial relationship of the components is desirably such that theillumination angle θ (FIG. 4 ) onto an item surface in the middle of thecamera's depth of field is 40 degrees or more. (The figure shows anillumination angle of 60 degrees. Some embodiments have illuminationangles of 75 or 80 degrees or more. If the camera has a straight-downorientation, the illumination source is this latter case is 15 or 10degrees or less angularly displaced from the camera lens, as viewed fromthe middle of the camera's field of view, and a mid-depth of fieldlocation.) Low angles diminish the surface illumination by a (1−cos θ)factor, requiring longer exposures.

(The angle of the light sources with respect to the camera optical axisis relevant as specular reflections from shiny objects often result insaturation of sensor pixels. The likelihood of seeing direct reflectionof a light source in the field of view is a tradeoff, as specularreflections are desired for detection of watermark signals embossed inplastics, but are not desired for reading printed watermarks from shinysurfaces. A balance can be achieved by assessing location of reflectionpoints when a mirror is placed on belt. Reflection points may be placedon the far top and bottom limits of the camera field of view.)

More on Feature Dimensions

As noted, one form of marking is a binary or sparse mark, in whichinformation is conveyed as an array of dots or other marking features.Such marks are generally made in respective cells of a lattice, withintervening cells left unmarked. The earlier-referenced examples use alattice of 128 rows by 128 columns of cells −16,384 in all.

Such cells are square in shape. Binary markings (whether printed, orformed by machining, laser, or other processes), in contrast, aretypically rounded, but may sometimes be square. Applicant naturallyunderstood that each mark should be confined to its respective cell.That is, the width of a mark should be less than or equal to the widthof its corresponding cell, so as not to intrude into adjoining cells.

Surprisingly, applicant found that this need not be the case. A mark canintrude into adjoining cells while still enabling satisfactory decoding.

To give a specific example, consider a mold used for thermoformingplastic. A sparse mark (e.g., a Type 2 binary watermark, as detailedearlier) is to be formed in the mold at a resolution of 75 waxels perinch. At this scale each waxel corresponds to a square area that is1/75^(th) of an inch on each side (13.33 mils). Such surface may beshaped (e.g., by machining or laser engraving) to form marks having theform of holes or depressions of circular cross section, e.g., withdiameters of 16 or 20 mils. Each such depression thus extends into allfour edge-adjoining waxels—by up to 1.33 mils in the case of a 16 milhole, and by up to 3.33 mils in the case of a 20 mil hole. Looked atanother way, 11% of a 16 mil hole's area falls within neighboring waxelcells, and 43% of a 20 mil hole's area falls within neighboring waxelcells. Nonetheless, when imagery depicting a plastic thermoform shapedby such mold is submitted for watermark decoding (e.g., using a knownwatermark decoder such as is described in U.S. Pat. Nos. 9,959,587 and10,242,434, and available in the Apple Store as the Digimarc Discoverapp for the iPhone), the watermark payload is correctly extracted. Thissame result is achieved even if the 50% or more of each mark's areafalls within adjoining waxels cells that the sparse pattern would leaveunmarked.

(To illustrate relative sizes, FIG. 5 depicts, in the upper right, ahole having a diameter equal to a waxel dimension of 0.0133 inches. Inthe upper left is a hole having a diameter of 0.02 inches. In the lowerright is a hole having a diameter of 0.016 inches.)

This discovered ability to use marking features that are larger thanwaxel sizes is important due to cost and manufacturability concerns.Tooling needed to make small features is typically more expensive andless durable than tooling needed to make larger features. Thus, it istypically more economical to produce items with, e.g., 0.02 inchfeatures than with 0.013 inch features. Such ability also enables use ofhigher WPI signal blocks, which, in turn, increases redundancy of signalcoding across a container.

As noted, laser marking can be used to form very fine features onsurfaces. FIG. 6 shows an enlargement from a sparse watermark pattern(Type 2, 150 WPI), in which each sparse dot is rendered as a corporatelogo. Faces, portraits, product or person silhouettes, and other graphicelements can be similarly utilized.

As just noted, the marks can be larger than the waxels—intruding intosurrounding waxels' territories that the algorithm which generated thesparse pattern would leave unmarked. In FIG. 6 , however, the intrusionisn't a fraction of a waxel. Rather, each of these logos is more thanthree waxels on a side. This is shown by the vertical and horizontallines of FIG. 6 , which show the centerlines of the columns and rowsdefining the lattice of waxel locations for this mark. Thus, more than50% of each logo's cross-sectional area overlays waxels other than thecentral waxels that is to be marked. (More like 75 or 80% of the logo'sarea is outside the intended waxel.) FIG. 5 shows, in the lower left, aportrait that can serve as a marking feature, overlaid on a lattice ofwaxel locations.

Thus, an aspect of the present technology concerns a physical itembearing a machine-readable code comprising a pattern of marking featuresthat collectively convey a plural-symbol message, where themachine-readable code is organized as a 2D lattice of edge-adjoiningcells, and a first of the cells is marked with a feature that extendsbeyond an edge of said first cell.

Briefly, a process for producing artwork like that shown in FIG. 6 caninvolve first generating a desired sparse dot pattern, using a tool suchas the Digimarc Plug-In for Adobe Illustrator. Next, the Image Tracefeature of the Illustrator software is used to turn the raster objectsrepresenting the sparse dots into a corresponding array of identicalvector boxes. Illustrator's Find-and-Replace scripting functionality isthen used to replace one of the boxes with a vector graphic (e.g., agraphic depicting a corporate logo). The script is then used tosimilarly replace all of the other boxes with the same graphic. Ifdesired, the resulting artwork can be converted into a pattern that canbe used to fill any region of any artwork, such as product packaging, byusing the Make New Pattern function of the Illustrator software. Theresulting pattern swatch can then be stored or distributed for lateruse.

Robustness Improvements

Since objects on the conveyor belt can be soiled, crumpled, and/oroverlay each other, it may be difficult to extract the watermark data.In particular, such phenomena tend to both attenuate the strength ofdesired reference and payload signals, and increase noise signals thatcan interfere with detection and reading of these desired signals.Various techniques can be used to increase the probability of readingthe watermark data in such circumstances.

One technique is to disregard certain frames of imagery (or certainexcerpts of certain frames of imagery) and to apply the computationalresources that might otherwise be applied to such imagery, instead, tomore intensively analyze other, more promising imagery (or imageexcepts). This technique can be used, e.g., when some or all of the beltdepicted in a captured image is empty, i.e., it does not depict a wasteitem.

Consider an embodiment in which image frames are captured at a rate of300 per second. About 250-300 blocks are processed from each frame, or75,000+ blocks per second. To control sorting, the system must operatein real time. With the belt moving 3 or 5 meters per second, and thediverters located just a few meters down the belt, the system has a verysmall interval in which to complete a very large processing task. Ifanalysis of a block (or frame) can be skipped, this time can instead beapplied to further-process other imagery.

One way to further-process imagery is to more intensively attempt todetect the presence of a watermark signal in the imagery, e.g., throughdetection of the reference signal. One way this can be done is to trydifferent 128×128 blocks (i.e., different block placements within theimage frame). In an illustrative embodiment, after pre-filtering (e.g.,by oct-axis filtering), a hundred or more different 128×128 pixel blocksare selected from each image frame. The selection can be random, or theblocks can be tiled in a uniform array, e.g., with each block having 50%overlap with the block to the left and with the block above. An FFT isthen applied to each of these blocks (optionally after windowing topreserve only the center 96×96 pixel patch, with surrounding pixelszeroed), and the resulting spatial frequency data is analyzed forpresence of the distinctive reference signal. The appearance of thisreference signal reveals affine pose parameters by which the watermarkblock is depicted in the captured imagery, as described earlier.

If such an estimate of pose parameters for a watermark block is reached(e.g., using the noted DLS procedure), the resulting affine transformdata can be used in a subsequent decoding operation, to identify waxellocations in the image data that should be sampled and provided to thedecoder. (In a particular embodiment, waxel locations may be sampledfrom an area of about 300×300 pixels centered on the block, to takeadvantage of payload signals that may be readable outside the boundariesof the pixel block from which the reference signal was found). From suchsample values the payload can then be decoded.

The number of blocks processed to attempt to detect the reference signal(e.g., 250-300 in an illustrative embodiment) is set to fully utilizethe available processors. That is, the number of processed blocks iscompute-bound.

If additional processing time is available (e.g., because an image frameor excerpt depicting empty belt is not being processed), then theprocess to find a reference signal can be performed more intensively.For example, 128×128 blocks may be more densely selected within theportion of the filtered image frame that does not depict empty belt.Perhaps from one of the densely-spaced blocks a reference signal will bedetected that would otherwise be missed, permitting additional watermarkdata to be extracted corresponding to an object depicted in acorresponding area of the frame.

A second way to more thoroughly (intensively) analyze imagery, ifadditional processing time is available, is to employ a different (e.g.,enlarged) set of DLS seed affine transforms—trying to find the referencesignal at poses not specified by the usual selection of seeds. Each seedtransform, in a particular embodiment, comprises a 2×2 matrix ofparameters, defining rotation, scale, and two dimensions of shearing(i.e., four dimensions in all) that describe a possible geometricpresentation of the watermark signal in the image block. The multitudeof seeds may normally sample a subspace of these parameters in a firstmanner, such as rotation between 0 and 359 degrees at one degreeincrements, scale between 0.5× and 1.5× in increments of 0.1, etc. Againthese parameters are normally chosen so that the processor(s) runs at100% utilization. If additional processing is available (because theimagery depicts vacant regions of the belt that needn't be processed),the affine transform parameter subspace can be sampled in a second,different, manner. For example, these parameters can span broaderranges, thereby increasing the range of affine presentations at which awatermark reference signal on an object depicted in the occupied regionof the image frame can be detected. Alternatively, these affineparameters can sample the subspace more finely (such as rotation atincrements of 0.5 degrees), thereby reducing the chance that theiterative DLS procedure will hone-in on a final pose estimate that issub-optimal.

Thus, for example, if the right half of an image frame is known todepict empty belt, then the number of DLS seeds employed in analyzingimagery from the left half of the image frame may be doubled, e.g.,using 2000 seed transforms instead of 1000 (or 20,000 instead of10,000). Processor utilization again reaches 100%, but such resource isapplied more intensively to a smaller set of pixels.

Thus, a method employing certain aspects of the technology concerns adigital watermark reading system that operates on an image captured acamera that is viewing a waste stream on a conveyor belt. The methodincludes identifying a first region in the image depicting an emptyregion of the belt, and in response, region, enlarging a set of affinetransform seeds employed by the digital watermark reading system insearching a second, different region of the image for a digitalwatermark.

Changing block boundaries and changing DLS seeds to increase thelikelihood of finding watermarks reference signals are but two of manyways that additional processing time can be employed to more thoroughlyanalyze imagery. Alternatively, or additionally, the extra processingtime can be applied to the payload decoding—rather than the referencesignal detection—operations.

For example, if the reference signal is detected in several nearby(e.g., overlapping) 128×128 blocks, watermark decoding may normally beattempted on only one of the blocks. In a particular embodiment, theimage frame is divided into eight sub-parts, and only one decode isattempted in each sub-part—based on the image block with the strongestgrid strength metric. However, if extra processing time is availablebecause not all of the frame merits analysis, the watermark decoding canbe applied to two or more such blocks, to increase the chances ofsuccessful watermark extraction.

In some embodiments additional processing time is employed to attemptcombining waxel data sampled from two or more different regions of aframe (or from different frames) to decode a single watermark payload.Such operation may not normally be undertaken, due to the short intervalwithin which all frame processing must be completed. But with additionaltime (e.g., gained because not all of the image merits processing), suchintraframe or interframe processing can be attempted.

Such processing assumes that the watermark reference signal has beendetected in each such region, revealing the poses with which the waxelpayload data is presented in the respective excerpts. Before combiningwaxel data from such excerpts, a check must be made that that tworegions depict surfaces of the same item. (As noted, watermark data istypically encoded in redundant, tiled fashion across the surface of anobject, so waxel data from different tiles can be combined. But only ifthe tiles are known to be from the same item.)

One way to check that two image excerpts, spaced apart within a frame,are from the same item is to perform a region-growing (blob detection)algorithm—extending out from one excerpt to see if the algorithm growsto encompass the second excerpt. Such methods are known to artisans,e.g., from the Wikipedia article entitled Blob Detection. If twoexcerpts appear to belong to the same item, as indicated by such aregion-growing method, then waxel data from one image excerpt may becombined with waxel data from the other excerpt, e.g., in weightedfashion in accordance with the grid strength metrics of the respectiveregions, as described earlier.

A way to check that two image excerpts, taken from two different imageframes, depict parts of the same item is to reverse the spatial movementthat the belt movement has caused between the two frames, e.g., byshifting the second image up or down or left or right in the frame by adistance corresponding to the time interval between the two imagecaptures. A spatial distance between the two excerpts—one original andone shifted—is then determined. If the center of one excerpt is within athreshold distance (e.g., 150 pixels) from the center of the otherexcerpt, then the two excerpts may be assumed to reliably depict thesame item, and waxel data sampled from the two excerpts may then becombined for decoding, as described earlier.

Alternatively, a region-growing algorithm can be applied to the itemregion depicted in the first image to determine the extent of aconnected blob of which it forms part. The second excerpt, shifted asdescribed above, is then examined to see if it overlies the connectedblob in the first image. If so, the waxel data in the two excerptslikely correspond to the same item, and again can be combined.

In both cases (i.e., excerpts spaced apart in a frame, or excerptsspaced apart in time) a correlation check can additionally oralternatively be performed. That is, a set of waxels that are depictedin common between the two excerpts are identified, and the patternformed by such +1/−1 waxel values in one excerpt is correlated againstthe pattern of such waxel values in the second excerpt. If thecorrelation exceeds an empirically-determined threshold value, thisindicates a likelihood that the two excerpts both convey the samepayload information, indicating they both likely depict the same item.This can be used as an independent, or a supplemental, test for whetherwaxel data from the two excerpts should be combined for decoding.

The foregoing, more intensive decoding efforts can be invoked ifcomputational resources are available due to part of the belt beingempty and not warranting watermark analysis.

A belt that is vacant across its width can be detected by a simplephoto-emitter/photo-detector pair that sends a light beam across thebelt (a “breakbeam” arrangement). If the beam is received on the farside of the belt with its full strength, it is highly unlikely thatthere is an intervening object on the belt. An array of several suchlight beams can be projected across the belt, collectively checking aswath several centimeters in length (e.g., the length of belt depictedin the captured camera imagery). The light beams can be low to the belt,such as a centimeter or two above the belt, below the top surface of anyplastic item that is likely to be conveyed by the belt.

FIG. 7 is a plan view looking down onto a belt, and showing a pluralityof LED emitters (with lenses, not shown) along the bottom side, andcorresponding photocells along the top side, defining breakbeams shownby dashed lines. If the bold rectangle is the camera field of view, withthe top to the right, it can be seen that the top 60% or so of thisimage frame can be disregarded, since no item is in this region of thebelt. Processing resources that would normally be applied to this partof the imagery can be applied otherwise.

This breakbeam method works only if the entire width of the belt is freeof an intervening object. A second arrangement is more flexible. In thisarrangement a laser line is swept (e.g., by a rotating mirror) acrossthe belt, from a projection system above the belt. A camera capturesimagery of the area along the belt at which the laser is aimed, wherethe laser line is expected to appear. If the line is missing or appearsdisplaced, this indicates an obstruction has intercepted the beam beforeit illuminated the belt (or has blocked the camera's view of the beam).That is, an item is present. As in the breakbeam arrangement, multiplesuch laser lines can be projected across the belt to localize whereobjects are present.

FIG. 8 is a plan view looking down at such an arrangement, with thelaser lines shown in dash. Again, the bold rectangle indicates a cameraview of the belt. The circle indicates an illustrative position of aviewing camera; the triangle indicates an illustrative position of thelaser projector. The dotted lines show how the container on the beltcauses the lines to appear displaced from their nominal positions, asseen by the camera. In locations where the laser lines appear straightalong their intended paths, the system can infer the belt is empty.Again, such regions of imagery can be disregarded, and associatedprocessing resources can be applied elsewhere. (FIG. 8A is a side viewof the same arrangement, with small black dots indicating where thelaser lines should fall if the belt is empty, and small squaresindicating how the laser lines are displaced in the presence of an itemon the belt.)

Many items on the belt may be crumpled or curved, so the straight laserlines may be distorted into non-linear traces when intercepted by suchitem surfaces. The angles and configurations (e.g., straight vs. linear)of these traces reveal information about the character and localorientation of object surfaces. For example, displaced lines that arestraight indicate they are illuminating a planar surface. A planarsurface on which two lines parallel lines are detected, with a linespacing wider than normally projected onto the belt, indicates thesurface is tilted away from the laser projector (and vice versa). Curvedlaser lines indicate projection onto a curved surface. Etc.

Knowledge of whether an item on a particular location of the beltpresents a curved or flat surface, or a surface tilted towards the laserprojector, or away, can be used to tailor the set of DLS seeds appliedin attempting detection of a watermark reference signal at suchlocation. One set of seeds can be used when a curved surface isindicated at a particular location; a second set of seeds can be usedwhen a planar surface tilted away from the laser projector is indicated;a third set of seeds can be used when a planer surface tilted towardsthe laser projector is indicated, etc.

The camera used in such embodiment can be dedicated to laser linedetection. Alternatively, imagery captured by another camera, such asthe camera used for watermark reading, can be analyzed for presence ofthe laser lines at their expected locations.

In a related arrangement, a depth sensing camera is used to image thebelt, producing a depth map image from which occupied and empty regionsof the belt can readily be distinguished. A suitable depth map camera isthe Intel RealSense 435, a stereo vision-based system with a globalshutter image sensor that can operate at speeds up to 300 frames persecond. Its frame captures can be synchronized with frame captures andflash illumination from the watermark sensing camera system. The brightflash helps reduce noise in the resultant depth data. Similar to thelaser line example just-discussed, the depth map image reveals whichitem surfaces are curved and whether they curve in the direction of belttravel or in the direction across the belt (or in between). It revealswhich item surfaces are planar, and the directions towards which suchsurfaces tilt. Such gross classification of surface type can be used toselect a corresponding set of DLS seeds that has been tailored for usewith such type surface.

Black and very dark items may be difficult to detect in the detaileddepth sensing arrangement, due to the low levels of light reflected tothe sensor, yielding noisy depth data. The depth data can be examinedfor excerpts with high local variance (i.e., high local noise), andwhere found can be treated as possibly indicating the presence of darkitems. Corresponding excerpts of the watermark imagery can then beanalyzed, irrespective of the absolute values of the depth dataindicated by the depth sensing system.

Similarly, specular reflection from shiny plastic surfaces can confusestereo vision depth sensing, since the location of the reflection in thefield of view can depend on the position from which a shiny surface isimaged. That is, the location of the specular reflection is not aninvariant landmark in the scene. Again, such confusion can yield noisysensor data, with one or more sudden shifts in the reported depth atlocations around the specular reflection. Again, regions in the field ofview having such aberrations in reported depth data can be treated aslikely having items that merit watermark analysis. (Put another way,only scene regions characterized by depth data consistent with thevarying distances to belt locations, with local noise below a fixedthreshold value, should be trusted as truly empty, and thus safe toignore.)

A third arrangement for identifying empty regions of the belt (or,similarly, identifying occupied regions of the belt) is based on beltocclusion.

A conveyor belt is initially homogenous in appearance, typically black.However, through use, the belt becomes scarred and stained. (Even whenthe belt is new there is a visible seam where the two ends of the beltare joined to form a loop.) These patterns repeat in captured imagery asthe belt loops around and reappear in the camera's field of view. If ascar or stain pattern normally reappears at intervals of about tenseconds, but at one such interval does not reappear, this indicates theview of the belt is occluded, i.e., by an item on the belt. By notingthe presence or absence of expected belt patterns in captured imagery,the system processor can discern whether a particular region of the beltis empty or occupied.

In a particular embodiment, the belt is “fingerprinted” when theconveyor is first turned on, and runs empty for a brief interval underillumination by the light system. As the belt travels, the cameracaptures image frames at the usual rate (e.g., 150 or 300 fps),“learning the belt” so to speak. The sequence of reference imagescaptured from a full cycle of the empty belt serves as a template fromwhich the depicted excerpts of empty belt can thereafter be recognized,e.g., by pattern matching, such as correlation.

In a brute force embodiment, a new image captured during wasteprocessing is correlated against each of the reference images gatheredin the initial fingerprinting phase of operation at different spatialalignments, to find pixel patches that exhibit high correlation. If thecaptured image depicts a portion of vacant belt, then pixels in thatexcerpt of the captured image should have a high correlation with acorresponding set of pixels in the reference imagery that depict thesame portion of belt. A map of correlation strength can be produced.Where the correlation strength exceeds a threshold value, the system caninfer that the corresponding region of the belt is vacant.

The brute force method need not be used. The speed of the belt is knownfrom a belt speed monitoring arrangement, so the same excerpt of beltreappears in the camera field of view at known intervals (e.g., aboutevery ten seconds, in the case of a 30 meter belt loop traveling at 3meters per second). Thus, the captured image need not be correlatedagainst all of the reference images. Instead, correlation can be checkedagainst only a dozen or so candidate reference frames, corresponding tothe excerpt of belt that is known to be within the camera field of viewwhen the new frame of imagery is captured (“proximate images”).

Moreover, the correlation operation need not consider all possible 2Dalignments of candidate reference images with the new image. The beltdoes not move much laterally; its movement is essentially in onedirection. So while the system can check for correlation between eachcandidate reference image and the new image at all possible spatialalignments in one dimension, it need check for zero or only a fewdifferent spatial alignments (e.g., offset by plus or minus up to adozen pixel columns) in the other dimension.

Such an arrangement is illustrated in FIG. 9 . A newly-captured capturedimage frame 91 depicts a dark region, in an area 92. A dozen or soproximate images of the belt were collected during one or more previouscycles of the belt, and their image data was collected into a dataset(here shown as a panorama image 93 for convenience) depicting nearbyareas of the belt. Included in the panorama 93 is an area 94 depicting aregion of the same shape and appearance—apparently a marking on the beltthat re-appears cyclically.

The imagery from the captured block 92 is correlated against imagery inthe panorama image 93 at a variety of spatial alignments (e.g., spacedapart by one pixel), as represented by the double-ended arrows. Onealignment (indicated on a frame-basis by the vertical hash marks 95)yields a peak correlation value. If this value is above a thresholdvalue, the newly-captured image data is not regarded as depicting newwaste items, but rather is classified as depicting something seenbefore—the belt. Such area of the newly-captured image frame 91 isconsequently flagged as empty.

The correlation value may be regarded as a match metric—indicatinglikelihood that the area of belt being analyzed is empty. The metric maybe refined by considering how “peaky” the peak correlation is. That is,whether the peak correlation is substantially above neighboringcorrelation values, or whether it is only modestly above. In onescenario the peak correlation value may be 0.9 (shown at the spatialalignment indicated by arrow 96 in FIG. 9 ), and the correlation valueat an adjoining correlation (e.g., offset by one pixel, indicated byarrow 97) may be 0.6. In a second scenario the peak correlation valuemay again be 0.9, but the adjoining correlation may be 0.2. The lattercorrelation is more “peaky” than the former because the difference inadjoining correlation values is larger. This latter scenario is morestrongly indicative of an empty area of belt.

In a particular embodiment, the peak correlation value is combined withthe difference between the peak correlation value and the adjoiningcorrelation value. One suitable combination is a weighted sum, with thepeak correlation value given a weighting of 1.0, and the differencebeing given a weighting of 0.5. In such case the former scenario resultsin a match metric of 0.9+0.5(0.3)=1.15. The latter scenario results in amatch metric of 0.9+0.5(0.7)=1.35. If the threshold is 1.25, then theimage area in the latter scenario is flagged as empty, whereas the imagearea in the former scenario is not (and thus is eligible for analysis toidentify watermark data).

In a further refinement, the peak correlation is compared against twoadjoining correlation values (i.e., correlations indicated at bothspatial alignments 97 and 98 in FIG. 9 ), and the larger difference isused in the weighted combination. If correlations are performed atoffsets across the belt, not just along its length, then there may befour adjoining correlation values. Again, the larger of the resultingdifferences can be used in the weighted combination.

The matching operation can be aided if synchronization marks are printedon the edge of the belt, e.g., at spacings on the order of a centimeter.If such marks are visible in a newly-captured image frame, then the beltdepicted in such frame can have one of only a few possible alignmentswith a frame of reference imagery, since the synchronization marksappear at the same positions on the belt in both depictions. This limitsthe search space of possible 1D alignments between the new and referenceimage data. (A small margin of error, on the order of a few pixels, maybe applied in the search for maximum correlation.)

Still further, the correlation need not be performed on full resolutionimagery. The imagery can be down-sampled in resolution and/or reduced inbit depth to reduce the computational burden. In a particular examplethe imagery is spatially down-sampled by a factor of four. In stillother arrangements, the images are oct-axis filtered before correlation,to simplify the task. Thus, derivative data produced from the newimagery can be compared with derivative data produced from the referenceimagery to determine empty/occupied regions of belt.

Yet further, the entirety of the new frame need not be considered inmatching with reference data. The new frame mostly depicts belt lengththat was depicted in the previously-captured frame. If the belt istraveling at 3 meters/second, and is being imaged at 300 frames persecond, then the belt advances just one centimeter between frames. Ifthe frame captures a depiction of 14 cm of belt along the direction ofbelt travel, then 92% of the belt depicted in the frame was depicted inthe prior frame. Thus, the correlation need focus only on the edge ofthe captured imagery that depicts belt newly-entering the camera fieldof view. In an exemplary embodiment, matching is performed only on thetop 10% or 20% of the new imagery.

Once a match has been found, at a particular spatial alignment, betweena newly-captured image and a reference image, this can simplifysubsequent searches for a match. That is, once a spatial relationship(offset) is found that yields maximum correlation between a new beltimage and a reference belt image, then nearly the same spatialrelationship should likewise exist between the next new belt image andthe next reference belt image. And so on for many future images in therespective sequences. The search for spatial alignments that yieldmaximum correlations for a new frame can thus be focused around thespatial alignment that yielded maximum correlation for a past frame.

(This discussion proceeds as if the reference imagery is a library ofdistinct reference images. Of course, such images can be stitched into asingle long reference image for the entire belt if desired.)

The belt fingerprint arrangement just-described can be self-learning.Imagery captured during one cycle of the belt can be correlated withimagery captured during one or more later cycles of the belt. Regionswhere the correlation is high (e.g., above a threshold value) betweensuch imagery indicate a consistent pattern on the belt—not a transientwaste item. If similar correlation is not found between such imagery andthe original reference imagery, this indicates that an item is presenton the belt, or that a new pattern (e.g., a new stain) has appeared onthe belt. If analysis of still later frames shows such pattern persiststhen reference imagery can be updated to include the new pattern.

In some embodiments, the initial fingerprinting of the belt by capturingimagery of the empty belt is not needed. Instead, the reference imageryis assembled on-the-fly from images of the belt carrying waste. Patchesof such imagery that are found to highly correlate between differentcycles of the belt can be inferred to depict the belt itself; not waste.Such patches are compiled in a data structure representing the compositeempty belt.

That is, a method employing certain aspects of the technology concernsdetermining appearance of an empty conveyor belt from images of the beltconveying items. Such method includes capturing images of the beltduring operation conveying items, where the items do not always coverthe belt. An image excerpt is identified depicting a portion of the beltin one image that correlates, with a correlation value exceeding athreshold, with an image excerpt captured during a previous cycle of thebelt. This identified excerpt is added to a data set indicatingappearance of the empty conveyor belt. The foregoing acts are repeatedto assemble a patchwork collection of image excerpts representingappearance of the empty conveyor belt.

By comparing newly-captured imagery with the reference imagery, areas ofempty belt can be detected, and computational resources can be directedfrom such areas towards other areas of the belt.

Applicant has discovered that fixed pattern noise in the camera system,e.g., due to processing variations among photodetectors in the sensor,or local aberrations in the lens, can interfere with the foregoingcorrelation operations—indicating a baseline of correlation when thereis none. To reduce such problem a dark frame subtraction technique canbe used. For example, at recurring intervals during operation of thesorting system a frame can be captured with none of the illuminationLEDs active (e.g., once or twice every five seconds). Given the shortexposure intervals, ambient light has been found to have nilillumination effect, and the resulting image is akin to that which mightbe captured with a lens cap over the camera lens. The pixel values fromthis dark image frame can be subtracted from counterparts in the othercaptured image frames to subtract the “fixed pattern” effect.)

That is, a method employing certain aspects of the technology involvesprocessing plural images of a conveyor belt, produced by an imagingsystem that captures images coincident with flashes of illumination,yielding plural arrays of illuminated pixel data. Occasionally an imageis captured without any flash of illumination, yielding a relativelydark frame array of pixel data. This relatively dark frame array ofpixel data is subtracted from each of the plural arrays of illuminatedpixel data.

Instead of fingerprinting the belt to sense where scar/stain patternsare revealed or occluded, the belt can be printed with a pattern toserve a similar effect. Indicia such as dots, circles, lines andcross-hairs may be used, which can rapidly be identified by simplepattern recognition algorithms. In a particular embodiment white circlesare printed across a black belt, as illustrated by FIG. 10 . The circlesare centered in one inch cells, within a virtual grid of such cellscovering the belt. Each circle is 0.75 inches in diameter, and is formedof a line that is 0.15 inches in width.

In this particular embodiment, imagery of the belt, e.g., captured forwatermark detection, is copied and converted into a binary image bythresholding and Gaussian filtering. Edges are next found, such as byapplication of the Canny algorithm. Finally, the edge points areanalyzed using a Hough transform to find circles of the known 0.75 inchdiameter. Grid cells in which such full circles are detected are knownto be empty regions of belt and thus are excluded from watermarkprocessing (or, inversely, grid cells in which full circles are notdetected are analyzed for the presence of the watermark referencesignal, etc.).

In another such arrangement the belt can be fabricated or treated withreflecting particles (glitter-like)—the specular reflections from whichindicate the camera is seeing bare belt, so no watermark extraction isneeded.

In an illustrative embodiment, successive image frames are capturedunder different spectral illumination (e.g., blue, red, or infrared).Features that are visible with one illumination may be invisible withanother. Groups of several (e.g., two or three) successive frames takenunder different illumination spectra can be spatially-registered andcombined to yield a composite greyscale image frame. A new compositeframe may be produced as each new frame is captured—with the new framereplacing the oldest component frame in the earlier composite frame. Insuch a composite frame no belt feature is likely to remain invisible.(The differently-illuminated frames may be given equal weightings toform the composite frame, or differently-illuminated frames may beassigned different weights. Spatial registration can be performed on thebasis of feature matching. Alternatively, the reference signal has beendetected in each of the frames, then combination can be basedregistration using the reference signals.)

The just-described fingerprinting arrangement can proceed on the basisof such composite frames. Additionally or alternatively, the detectionof watermark reference signals and/or reading of payload data can beperformed on such composite frames. (So, too, can artificialintelligence-based recognition.)

While time is one computational resource that can be reallocated ifempty belt imagery is detected, there are others, such as memory andprocessor cores (more generally, hardware resources). By being able toallocate hardware resources away from where they are not needed to wherethey are, faster and better results may be obtained.

Another circumstance—other than belt emptiness—in which computationalresources can be conserved is when the item occupying a region of beltis known to not need (further) watermark processing. This can happenbecause, at the high frame rates typically involved, there may be adozen or so images depicting each item as it passes across the imageframe—each depiction being advanced about 1 cm from the previousdepiction. If a watermark is read from an item in one frame, and theitem will be depicted in the next ten frames too, that the regionoccupied by that item can be ignored as the location of such regionsteps linearly across the following frames. (Additionally oralternatively, blocks adjoining that region can be analyzed insubsequent frames to discover the extent of the watermarking, and thuslearn more information about the extent of the item. Such analysis canbe shortcut since pose data from the earlier watermark read is astarting point for estimating pose data for watermark reads insubsequent frames—again conserving processing resources, enabling otherregions to be more intensively analyzed.)

Thus, a method employing certain aspects of the technology can includecapturing a sequence of images with a stationary camera that views amoving conveyor belt carrying items in a material stream, where theitems in the material stream advance a fixed distance between imagecaptures. In one of the captured images, an attempt is made to read a 2Dmachine readable code from imagery corresponding to a first region onthe belt, and this attempt is successful, yielding payload data. In anext of the captured images, no attempt is made read a 2D machinereadable code from imagery corresponding to a second region on the belt,where the second region is the first region advanced by the fixeddistance. Computational resources saved by not attempting to read a 2Dcode from the second region are applied to attempts to read a 2D machinereadable code from other regions of the second captured image.

More generally, it will be recognized that one aspect of the presenttechnology is determining how intensively to analyze image data in anattempt to find or recover watermark information, based on how much ofthe image data depicts empty or known belt

Returning to DLS seeds, a further optimization is to tally how ofteneach of the DLS seeds succeeds in yielding a successful decode. That is,count how often a successful watermark decode operation is based onreference signal pose parameters iteratively derived from each of theseeds, e.g., in the form of a histogram or other data structure. Suchdata can be compiled over vast numbers of image frames (e.g., tenmillion frames, which corresponds to about 10 hours of operation, at 300frames/second). Seeds that yield successful watermark decodes aremaintained. Seeds that don't yield successful decodes are discarded.Seeds can be applied in order of their success rates, so that ifreference signal detection time must be curtailed for a block, the mostpromising seeds will have been applied first.

New seeds with different affine transforms can be introduced when othersare discarded. The new seeds are similarly tested over millions of imageframes. (The new seeds can extend the four-dimensional envelope ofsampling subspace into new regions, or can more densely sample theexisting sample subspace.) Over time an optimized set of seeds evolves,comprising only seeds that have a history of success.

Seeds that were earlier discarded may be tried again by the systemhours, days or weeks later, on the chance that the composition of thewaste may have changed so that seeds which formerly failed to lead tosuccessful decodes may later be found to do so. The system therebylearns and adapts its operation, so that the set of seeds that is usedthis week is commonly different than the set of seeds that were usedlast week.

Thus, a method employing certain aspects of the technology concernsdetecting coded markings on surfaces of items depicted in differentimages, where the coded markings each includes a common reference signalthat has different appearances in the different images depending on theposes with which the surfaces are depicted in the images. The poses areeach characterized by a respective set of pose parameters. The methodincludes receiving seed data including plural different sets of poseparameters, and receiving an image. Different sets of the poseparameters of the seed data are tested to determine which particular oneof the tested sets of pose parameters best describes the appearance ofthe reference signal within the received image. A data structure, suchas a pose success histogram, is updated to indicate which particular oneof the tested sets of pose parameters best described the referencesignal appearance. This is repeated a thousand or more times withdifferent images, adding data to the histogram. One set of poseparameters is then identified for removal from the seed data, based onthe data in said pose success histogram (e.g., the historicallyleast-successful set of pose parameters), yielding modified seed data. Afurther image is then received, and different sets of pose parametersare tested from this modified seed data to determine which particularone of the tested sets of pose parameters best describes the appearanceof the reference signal within the further image. This determined set ofpose parameters is then refined to still better describe the appearanceof the reference signal within the further image. A payload is thenextracted from the further image using the just-refined pose parameters.

An increase in robustness can further be achieved by using the imagesensor (e.g., the Sony IMX425) in 12-bit mode rather than the usual8-bit mode. This provides two additional least significant bits, and twoadditional most significant bits.

The two additional least significant bits offer two bits of greaterprecision, by which very small variations in light reflection (which arenot uncommon in watermark signaling) can be discerned. These extra bitssometime make the difference between a reference signal being detectedfrom a block of imagery or not, or between a payload from beingsuccessfully decoded or not.

The two additional most significant bits extend the saturation limit ofthe sensor. Features that produce identical 255-value signals in an8-bit image representation may be distinguished as different, againleading to gains in reference signal detection and watermark payloadrecovery. Additionally, these most significant bits enable signalrecovery from item surfaces that extend high above the belt. Suchsurfaces are more brightly illuminated due to their proximities to thelight source. Features in such regions that are washed-out by saturationin 8-bit sensors can contribute useful reference signal and payloadsignal information when 12-bit mode is used.

For similar reasons, sensors with 14- and higher-bit capabilities canlikewise provide still further performance improvements.

Watermark extraction must typically occur in essentially real-time, ifthe information thereby obtained is to be used to control sorting. Someinformation, however, is not so time-critical. One is collection ofstatistics, such as counts of different products produced by aparticular brand (e.g., cola, diet cola, and root beer). Another istracking return of serialized items. Imagery can be collected as thebelt is running, and archived for later, offline (perhaps cloud-based)analysis to extract this and other information that is not requirednear-instantly for sorting.

Additional Details

Applicant conducted various tests on thermoformed plastic surfaces,formed from molds marked with signaling patterns of differentvarieties—both continuous (continuous tone) and sparse. FIG. 11 depictsone such test sample—a container lid made of recycled PET, which wasshaped to include a multitude of test patches of sparse dot patterns (asdetailed in publication 20190332840), with different dot densities anddot sizes.

Surprisingly, applicant found that more signal (i.e., more plasticdeformation in accordance with watermark signal) does not lead to moredetection. Instead, applicant found that sparse markings detect morereliably than continuous markings. Moreover, applicant found that fewerdots in the sparse pattern lead, to a point, to more robust signaldetection.

One method of assessing signal robustness is to capture imagery of atextured surface, and then add noise to the imagery before attemptingdata extraction. The amount of noise that an image can tolerate, whilestill yielding better than 50% decoding success, is a metric of signalrobustness. A related method proceeds similarly, but attempts watermarkreading in the presence of increasing levels of gaussian blur, todetermine at what blur level 50% decoding success is still achieved.

Such techniques were applied to a great number of samples, variouslyconfigured with different parameters (e.g., the percentage of availablelocations that are marked, the size of the dot at each marked location,and the embossing depth). An illustrative set of 20 test samples (4 rowsby 5 columns) is shown in FIGS. 12A and 12B. (These figures comprise asingle table of 4 rows and 5 columns when placed side by side. Thepatterns are not accurately depicted at this scale due to reprographiclimitations, but the illustrated patches give a gross sense as todifferences.)

The fourth sample in the second row (outlined with a rectangle in FIG.12B) has a lower legend that ends with 200wpi_600dpi_ds3_c1. Thisindicates the sample has 200 watermark elements per inch, and isrendered with a resolution of 600 dots per inch. The “ds3” indicatesthat each mark approximates a circle of diameter 3 pixels at therendering resolution, e.g., is a 3×3 square array of marks. The “c1”indicates that 10% of available marking locations are actually marked,or 5% of all locations. (The “available” marking locations are regardedas being half of the total number of locations, since the densest sparsemarking is typically a checkerboard pattern with every-other locationmarked.)

Legends underneath many of the samples are truncated due to spaceconstraints, e.g., lacking the “c1” data. Others abbreviate “c1” as“dd10c3” (and “c2” as “dd20,” and “c3” as “dd30,” etc.).

All of these patches have an embossing depth of 0.5 mm, although otherdepths were also tried.

FIG. 13 illustrates some of the robustness data gathered by addingGaussian noise to imagery of various samples, and attempting decoding.Robustness data for the 20 samples of FIGS. 12A and 12B are plottedbeginning mid-way along the horizontal axis, with each pair of barscorresponding to a single sample, read across row by row. Data for therectangle-outlined sample of FIG. 12B are indicated by thedownward-pointing arrow at the top of FIG. 13 . The left bar of eachpair indicates relative decoding robustness for a thermoformed plasticsample positioned on a neutral grey background, lying flat on a conveyorbelt, embossed side up. The right (and routinely-taller) bar of eachpair indicates robustness for the plastic sample similarly oriented butelevated three inches above the conveyor belt.

A few of the best-performing samples are indicated by the callout boxesin FIG. 13 . Each box specifies, for the corresponding pattern, (a) thefraction of available locations that are marked (e.g., “DD10”), (b) thedot size (e.g., “DS3”), (c) the embossing depth, and (d) the WPI. (Theembedding protocol, V2 or V3, is also noted. These protocols correspondto the Type 2 and Type 3 watermarking algorithms, reviewed above, bywhich mark locations are selected, and are further detailed inpublication 20190332840 and pending application Ser. No. 16/849,288,filed Apr. 15, 2020.) Interestingly, the “DD10” (“C1”) patterns, forwhich only 5% of the surface area is marked, were routinely among thebest performers, with certain of the “DD20” patterns also performingwell. Generally speaking, clumping more dots together to form marks(e.g., 3 instead of 2) increased robustness, as did providing moreisolation between marks (a corollary to marking less of the surfacearea).

The data of FIG. 13 is presented in tabular form in the following table:

PATCH ON BELT 3″ ABOVE BELT ROW = 1; COL = 1 0 0 ROW = 1; COL = 2 0 8.91ROW = 1; COL = 3 3.96 14.42 ROW = 1; COL = 4 5.2 14.43 ROW = 1; COL = 50 0 ROW = 2; COL = 1 2.03 15.73 ROW = 2; COL = 2 4.82 8.21 ROW = 2; COL= 3 0 16.83 ROW = 2; COL = 4 17.51 36.21 ROW = 2; COL = 5 0 26.99 ROW =3; COL = 1 0 17.05 ROW = 3; COL = 2 4.43 20.82 ROW = 3; COL = 3 0 7.24ROW = 3; COL = 4 0 15.9 ROW = 3; COL = 5 7.22 19.12 ROW = 4; COL = 14.63 15.67 ROW = 4; COL = 2 0 11.97 ROW = 4; COL= 3 0 15 ROW = 4; COL =4 0 9.28 ROW = 4; COL = 5 0 14.81

Tests of robustness in the presence of blur yielded similar results.

Testing also found that plastics marked with sparse patterns, ratherthan continuous patterns, yielded better decoding robustness in thepresence of noise and blur.

FIGS. 14 and 15 show excerpts of some sample imagery captured from threeinches above the conveyor belt. FIG. 12 depicts a sample marked at“DD10”, and FIG. 13 depicts a sampled marked at “DD30.” FIGS. 14A and15A are counterparts that have been inverted, and contrast-altered, tobetter depict certain of the differences. The thermoform of FIGS. 14(and 14A) has about 5% of the area marked, with 95% of the plasticsurface following its original, nominal flat contour. The thermoform ofFIGS. 15 (and 15A) has about 15% of the area with marked, with 85% ofthe plastic surface following its original flat contour.

As in other arrangements, the information encoded in the pattern caninform a recycling system as to the type and use of the plastic, and itspreferred disposition. For example, the encoded information can identifythe manufacturer and the product (for reduced extended producerresponsibility, or EPR, fees), whether the item was used for food ornon-food packaging, whether the plastic is recyclable or composable, thecomposition of multi-layered packaging, etc.

While the just-discussed data particularly concerns thermoformedplastics, the same performance phenomena (less dots, bigger dot sizes,and more dot isolation, all typically yield better robustness) carriesforward to other plastic shaping technologies, such as laser shaping.Laser shaping also makes plastic serialization practical, i.e.,embedding a different signal in each different instance of, e.g., a runof 100,000 soda bottles. A payload field may be incremented, from onebottle to the next, and a corresponding pattern generated (e.g.,according to one of the algorithms detailed in publication 20190332840and pending application Ser. No. 16/849,288, filed Apr. 15, 2020) andprovided to control the laser marker.

Both laser engraving and laser etching can be used to mark and serializeplastics. (Some artisans use the term “engraving” to mean cutting acavity into the surface, typically by vaporizing the plastic, and use“etching” to refer to heating the top surface of an article to the pointthat its appearance changes but not to the point of vaporization.Applicant does not observe a strict distinction, but commonly uses theterms interchangeably. Likewise with laser “embossing.”)

Different lasers yield different effects with different substrates. Forexample, a 10600 nm laser (CO2), when applied to PET, is prone to yieldan engraving effect, with material vaporized and the remaining surfacemolten/congealed, and left chaotic from bubbling. This can make suchlasers ill-suited for use in marking PET bottles with line art patterns(e.g., Voronoi and Delaunay patterns) due to potential breach of thebottle sidewall, which may be only 10 mils in thickness. In contrast, acomparably powered and focused laser that is tuned to 9300 nm is foundto mark PET surfaces with a surface frosting, with minimal vaporization.The frosting provides good visual contrast—both in clear PET and incolored PET (e.g., black). The difference between lasers of such similarwavelengths is due to PET's radically-different absorption (extinction)at different wavelengths. (Of course, in some contexts, the deeper andmore chaotic effect of a CO2 laser suits the application.) Otherplastics (HDPE, PP, etc.) exhibit similar wavelength-dependentabsorption variation.

Much laser marking of plastic is done using so-called fiber lasers,which use a flexible optical fiber to both generate and deliver thelight energy, enabling high accuracy at relatively low cost. Such lasersare available for a variety of wavelengths, including 10600 and 9300 nm.

FIG. 16 shows a CO2 laser-marked PET bottle (contrast-adjusted forreproduction purposes). FIG. 16A is a close-up taken from FIG. 16 , alsocontrast-adjusted.

Combinations of Item Identification Technologies

The technologies detailed herein can be used in conjunction with otheridentification technologies to advantageous effect. One such alternativetechnology involves spectroscopy, such as near infrared (NIR)spectroscopy.

Spectroscopy systems commonly determine a spectral signature of aplastic resin by identifying the resin's optical absorption(transmittance) at a variety of different wavelengths. Some systemscorrelate such a spectroscopy signature with reference signatures ofknown plastics to determine which known plastic provides the best match.Other systems use machine classification techniques, such as neuralnetworks or support vector machines, to similar effect, determiningwhich known plastic has spectral absorption attributes that most closelymatch those of a container being analyzed. Related techniques rely onfluorescence of plastic items under hyperspectral illumination, e.g.,due to fluorescing additives included in the plastic resin. Again,resulting spectral data is compared against reference fluorescence datafor known varieties of plastic. All such techniques are here referencedunder the term spectroscopy.

Some such methods are further detailed in U.S. patent publicationsincluding U.S. Pat. Nos. 5,703,229, 6,433,338, 6,497,324, 6,624,417,20040149911, 20070296956, 20190047024 and 20190128801.

An exemplary material sorting facility may include a first detectionsystem adapted for identifying items by watermark data, and a seconddetection system adapted for identifying items by spectroscopy. Eachsystem uses a different camera system, although this is not required.Typically, the camera system used by the first, watermark reading systemis earlier in the processing line relative to the spectroscopy camerasystem, to permit additional time for the watermark signal to beidentified and recovered from the captured imagery before items travelto the region where sorting diversion (e.g., by forced air, or“blowout”) takes place. FIG. 17 shows an illustrative diagram. Eachidentification system is shown with an associated database, which in thewatermark case is a resolver database that provides item attribute dataassociated with different watermark payloads, and in the spectroscopycase is a reference library of known absorption/fluorescencepatterns—associating each with plastic identification data.

Each frame captured by the watermark reading camera system is taggedwith a timestamp indicating its time of capture. Within each frame, anyblock or sub-block from which a watermark decoding succeeds is taggedwith identification data (e.g., the decoded payload, or a plastic typeobtained from a database based on the decoded payload, or a particulardiverter that should be activated to deflect the item from the materialflow, etc.). Given the fixed geometry of the camera relative to thebelt, each position within a captured frame corresponds to a uniquespatial belt position. The speed of the belt may be regulated at a knownspeed. Or the belt speed may be measured by tracking the rate at which avisible feature on the belt processes through camera frames captured atknown times. Knowledge of the time an image frame was captured, the beltspeed, and the position of an identified item block within the frame,enables future positions of the item to be predicted. The location ofthe diverter apparatus is also known, as are its timing characteristics.This enables the diverter apparatus to be activated by the sorting logicprocessor at the instant at which the identified item is properly inposition for ejection by the diverter.

Sometimes the watermark-based system and the other (e.g.,spectroscopy-based) system will recognize an item, but indicate slightlydifferent spatial positions for it, leading to different diversionparameters (e.g., which air jet and at which instant). One approach isto then average the different spatial positions, and to base thediversion parameters on the average. Alternatively, one system may begiven priority in determining the diversion parameters, with any variantparameters indicated by the other system simply ignored. Such prioritymay be fixed, or may depend on data collected by the systems. Forexample, if the watermark system reports detection of a single watermarkblock, then it is known that such detection occurred at just a smallphysical excerpt from what may be a much larger item (e.g., a watermarkblock may be less than an inch on a side, and yet such block may appearon a liter drink bottle that is 38 cm tall). Relatively little is thusknown about the extent and orientation of the item. In such case, thediversion parameters indicated by the other technology (spectroscopy)may control diversion. In contrast, if multiple watermark blocks (whichmay be overlapping watermark blocks) are decoded from an item, then morecomplete data about the extent and orientation of the item is available,in which case the diversion parameters indicated by the watermark systemcan control diversion. This is illustrated by FIGS. 18A-FIG. 18D.

Referring to FIG. 18A, if a drink bottle is identified by data collectedat one position on the belt (e.g., by a solitary block of digitalwatermark data), and the dimensions of the bottle are known (frommetadata lookup based on an identifier decoded from the watermark) to be23.5 cm in height by 6.5 cm in width, then the bottle—if not crushed—canoccupy space anywhere within a circle 23.5 cm in diameter, centered onthe identified location. If, however, the bottle is identified fromwatermarking detected at two patches that are 16 cm apart, this distancebetween the detection locations constrains possible areas on the beltoccupied by the bottle; a region smaller than a circle of 23.5 cmdiameter can be determined. This is shown by FIGS. 18B-18D. The solidline in FIG. 18B shows one possible position of a 23.5×6.5 cm boundingbox that encompasses both locations. The dashed line shows another. FIG.18C shows another such pair of bounding boxes that encompass bothlocations. In the aggregate, the geometrical constraints imposed by thetwo detection locations, and the known dimensions of the bottle, definean hourglass-like shape where the bottle can lie on the belt, as shownin FIG. 18D. Thus, the greater the number of watermark block detectionsfrom an item, the greater the information about its extent andorientation, and the more trustworthy such information becomes as abasis for diversion parameters, relative to spectroscopy or otheralternative(s).

In still other arrangements a laser-based system for identifyinglocations of items on the conveyor belt is employed in conjunction withdata from one or more of the item identification systems, to controldiversion of items of the belt. In yet other arrangements the system cangive item-locating precedence to whichever of the two systems isphysically-closest to the diverters—reasoning that the item location onthe belt may have changed (e.g., due to tumbling) between its sensing bythe two systems.

Spectroscopy systems identify plastic type, and watermark systemsidentify plastic type as well as other item attribute data stored in theresolver database (information that is typically stored there at thetime of the item's creation, or before). Some sorting, however,desirably involves criteria not known at the time of the item'screation, but rather describes the item's state on the conveyor. Is itdirty? Does it have a cap? Is it crumpled? Etc. Such factors may betermed state attributes. Machine learning techniques (sometimes termed“AI,” “ML,” or deep learning, often implemented with convolutionalneural networks trained using gradient descent methods) can be employedon the processing line to gather such state information. The presenttechnology includes joint use of AI techniques with watermark and/orspectroscopy techniques to increase the accuracy and granularity withwhich items are identified for sorting. (Prior art AI techniques thatare suitable for such applications are detailed, e.g., in U.S. patentpublications 20180016096, 20180036774, 20190130560 and 20190030571 toAMP Robotics, Inc., CleanRobotics, Inc., and ZenRobotics Oy.)

If two analysis systems (e.g., watermark and spectroscopy and/or AI) areused to identify a single container attribute, such as plastic resintype, they may sometimes give conflicting outputs. This can occur, forexample, if a spectroscopy system encounters an object of unusualplastic composition for which it does not have a corresponding referencesignature, or if an AI system hasn't been sufficiently trained torecognize a particular variety of container. Such a system may identifythe best match as being to a different, incorrect, plastic. Conflictingoutputs can also occur if a company changes the resin composition of aproduct container without providing updated plastic information to awatermark resolver database entry associated with that product'swatermark payload.

When conflicting outputs occur, the sorting system can treat the objectas unidentified, and not divert it to any resin-specific destination.Alternatively, the system may include one or more rules to arbitrate orreconcile among conflicting outputs. For example, a sorting logicprocessor (FIG. 17 ) can receive outputs from the two systems and beconfigured to apply a rule such as: IF watermark reading indicates aplastic type for which a spectroscopy system does not have a referencesignature (perhaps polyoxymethylene), THEN the watermark-based resinidentification is to be given precedence (i.e., the object is to besorted in accordance with the watermark-indicated identification); ELSEthe spectroscopy-based resin identification is to be given precedence.

Spectroscopy systems typically fare poorly in identifying black and darkplastics, due to the lack of reflected illumination. If an object isrecognized to be black (or dark) in reflectance, and thespectroscopy-based system outputs a resin identification that conflictswith a resin identification provided by the watermark-based system, thena rule in the sorting system processor can cause the watermark-basedresin identification to be given precedence for sorting purposes, withthe spectroscopy-based identification being disregarded. (Black or darkobjects can be recognized as such from imagery—which may be collected bythe watermark system camera, the spectroscopy system camera or anothercamera—when the belt is illuminated with multispectral light. Suchobjects can be characterized by low average pixel intensity, e.g.,having an average pixel value below a threshold value, such as 30 in an8-bit image.)

The sorting logic processor may thus have a rule that (a) IF thewatermark system identifies an object as being composed of a resin thatthe spectroscopy system is also capable of identifying—but did not, and(b) IF the object is not dark (e.g., if it is light or transparent),THEN sort the object in accordance with the spectroscopy-indicated resin(reasoning a brand may have changed the object composition, and theresolver database has not yet been updated); ELSE sort the item inaccordance with the watermark identification.

Since spectroscopy and AI identification systems are probabilistic, suchsystems can produce data indicating confidence in their outputidentifications. For example, if measured spectral absorption data foran item closely-matches the reference absorption data for a particularplastic (e.g., correlation in excess of 0.9), then the identificationcan be given a high-confidence grade. If correlation is between 0.6 and0.9, the identification can be given a mid-confidence grade. Ifcorrelation is between 0.3 and 0.6, the identification can be given alow-confidence grade. (In some embodiments, correlation is calculatedand used as a confidence grade; in other embodiments a neural networkderives a confidence value between 0 and 1.) The rules to arbitratebetween conflicting resin identifications can depend on such confidencemetrics. For example, precedence may be given to aspectroscopy-indicated resin over a watermark-indicated resin, in therule set given above, only if the spectroscopy confidence is high- ormid-grade. That is, the rule logic becomes: (a) IF the watermark systemidentifies an object as being composed of a resin that the spectroscopysystem is also capable of identifying—but did not, AND (b) IF the objectis not dark (e.g., if it is light or transparent), AND (c) IF thespectroscopy system indicates a confidence of high or mid, THEN sort theobject in accordance with the spectroscopy-indicated resin, ELSE sortthe object in accordance with the watermark-indicated resin.

Some containers are “sleeved” by a thin layer of plastic (e.g., a shrinklabel that wraps a bottle) having a plastic composition different thanthat of the underlying container. When a watermark is decoded from asleeved bottle, the watermark metadata can indicate the presence of thesleeve layer and its plastic composition, and also indicate the plasticcomposition of the underlying bottle. If the underlying bottle istransparent PET, for example, watermark identification permits this factto be determined and the bottle diverted to a bin with other transparentPET bottles, even if the bottle is sleeved in an opaque, colored film ofanother plastic type. (The sleeve may later be removed and separated ina float tank, since common labels such as polypropylene and polyethylenehave a specific gravity less 1.0 and thus float, while PET has aspecific gravity greater than 1.0—typically 1.4—and thus sinks.)

Here again, sleeving is a factor that can be employed in reconcilingdifferent resin identifications indicated by a watermark (WM) and other(e.g., spectroscopy) identification systems. A sample system may applythe following sequentially-applied rules of reconciliation logic:

-   -   IF the watermark indicates a sleeved container, THEN sort per        watermark indication of underlying plastic, and end;    -   IF the watermark indicates the container is composed of a resin        that the spectroscopy system is also capable of identifying—but        did not, AND IF the container is not dark, AND IF the        spectroscopy system indicates a confidence of high or mid, THEN        sort the object in accordance with the spectroscopy-indicated        resin, and end;    -   ELSE sort the object in accordance with the watermark-indicated        resin.

A variant of this process is shown by the flow chart of FIG. 19 .

That is, a method employing certain aspects of the technology caninclude receiving imagery depicting a container on a conveyor, where thecontainer comprises a first, substrate material, wrapped by a second,sleeve material. A 2D code depicted on the sleeve material is decoded toyield a payload, which indicates a plastic type of the first, substratematerial. The container is then diverted into a repository with othercontainers comprised of said first substrate material, through use ofthis payload.

(Some items are composites of plastic with non-plastic materials.Examples includes certain disposable coffee cups, which have a plasticinterior, and a fibrous paper exterior. The fibrous material providesthermal isolation from the cup's hot contents, while the plasticinterior provides watertightness. The non-plastic exterior of sucharticle can be watermarked—by printing or texturing—to convey acontainer code which indicates the resin composition of the interiorplastic. When encountered in a material flow, such article can therebybe sorted for recovery of the plastic interior, based on imagerydepicting the exterior non-plastic medium.)

As indicated, sorting can be based on a combination of item attributes,rather than on plastic type alone. In one such system, spectroscopy isused to identify the object's plastic type. Watermark decoding isemployed to determine other object information, such as whether acontainer was used for food or non-food. A PET food container can thenbe diverted to a bin for food containers made of PET (bin #1), while aPET container for tennis balls can be diverted to a bin for non-foodcontainers made of PET (bin #2). The contents of the first bin can besent to a processor for recovery of food-grade PET recyclate, andcontents of the second bin can be sent to a processor for recovery ofnon-food-grade PET recyclate.

In another such arrangement, watermark decoding is used to identify botha container's plastic composition and its food/non-food status. A secondsystem, using a trained AI classifier, visually grades containers asappearing to have more or less than a threshold degree of contamination(e.g., external soiling or residual contents within). Containers thatare judged by the watermark system to be of PET resin and food-type, andwhich are judged by the neural network classifier to have less than thethreshold degree of contamination (“clean”), are diverted to one bin.Containers that are judged to be PET and food, but are classified asdirty, are diverted to a second bin. Containers that are judged to bePET and non-food, and are classified as clean, are diverted to a thirdbin. Containers that are judged to be PET and non-food, and areclassified as dirty, are diverted to a fourth bin. Four further bins maybe allocated to HDPE containers of the various types. Etc. Each bin ofcontainers can then be processed separately, assuring that recyclates ofthe highest possible purities and economic values are produced.

Other container attributes on which sorting can be based, jointly withplastic type and/or other factors, include color, whether HDPE isnatural or pigmented, whether the plastic is virgin or recycled, whetherthe container is sleeved, whether the container is a multi-layerstructure, age and/or refill count of a serialized refillable container,and whether a cap is present on a container.

As noted, the presence of a cap on a container is an item of metadatathat an AI system can be trained to discern. To assure the highestpurity recyclate, some recycling processors want to avoid accepting PETbottles with caps attached, as the caps may be made from a different,contaminating plastic. In one particular arrangement, such a cappedbottle-discerning AI system is positioned before the watermark readingsystem, and the former communicates map data to the latter, indicatingthose positions on the approaching belt where capped bottles have beenidentified. The watermark reading system can then ignore correspondingareas of imagery captured by the watermark system camera(s). Thewatermark reading system, or the AI system, can flag the capped bottle'slocation on the belt so that the bottle is ejected into a collection binwith other capped bottles. Alternatively, the capped bottle can bepermitted to travel the length of the conveyor and be discharged withunsorted items. The computational effort saved by the watermark readingsystem in not processing imagery depicting an item unsuitable forrecycling can be applied elsewhere, as discussed earlier in connectionwith empty regions of the conveyor belt.

More generally, an AI system can be trained to classify a dozen or morecategories of items likely to be encountered on the belt, and labelcorresponding areas on a map of the belt. FIG. 20 shows such anarrangement, in which different areas (each identified by a pair ofcorner coordinates) are respectively identified as having an aluminumcan, a capped plastic bottle, an uncapped plastic bottle, a black tray,and a wad of paper. One technology for such spatial labeling of multipleitems within an image frame employs so-called “R-CNN” techniques(region-based convolutional neural networks), such as that by Girshickdetailed in “Fast R-CNN,” 2015 IEEE Conference on Computer Vision andPattern Recognition, pages 1440-1448, and elaborated in Girshick's paperwith Ren, et al, “Faster R-CNN: Towards Real-Time Object Detection withRegion Proposal Networks,” arXiv preprint arXiv:1506.01497, Jun. 4,2015, and in patent document US20170206431.

In an illustrative plastic recycling system, there is no need to attemptwatermark decoding of an aluminum can, or a capped bottle, or a wad ofpaper. The AI system provides map data reporting these objects and theirlocations to the watermark reading system, which then can disregardthese areas and focus its analysis on other areas. The watermark readingsystem can additionally, or alternatively, limit its analysis efforts tothose regions of the belt indicated, by the AI system, as occupied bythe uncapped bottle and the black tray. Such an arrangement is shown inFIG. 21 .

Still further, such an AI system may be trained, through use of labeledtraining images and gradient descent methods, to identify locations offold contours in depictions of crushed plastic objects, and/or theless-disturbed surfaces between fold contours. Again, such map data canbe passed to a watermark reading system, which can sample image blocksfor analysis on the less-disturbed surfaces between the fold contoursand can apply less or no analysis efforts on regions encompassing thefold contours (where watermark reading may be less successful).

The map data generated by the AI system and communicated to thewatermark system can be specified in terms of pixel locations within theAI system camera field of view. Alternatively, such pixel locations canbe mapped to corresponding physical coordinates on the conveyor belt(such as at a position 46.5 feet from a start-of-belt marker, and 3inches left of belt center line.) Given a known belt speed and a knowndistance between the AI and watermark system cameras, the mapping ofeither to corresponding pixel locations within the watermark systemcamera field of view is straightforward.

Some or all of the data obtained by watermark decoding may not be usedfor sorting, but rather is used for statistical or other analysis. Forexample, a soft drink brand may bottle all of its various 12-ouncedrinks (cola, root beer, iced tea, etc.) in containers of identicalplastic composition, all of which are sorted into a common bin forrecycling. But watermark data printed on the container labels, ortextured on the container surfaces, allows the different products to bedistinguished. Such data can be compiled into statistical reports, e.g.,tallying counts of each different product processed by the sortingfacility, by day, week, month, etc. Additionally, or alternatively, thefacility can tally waste delivered from different sources (e.g.,different neighborhoods, sporting stadiums, etc.) as separate batches.For each batch a report can be generated, e.g., counting the number ofeach product within a brand family, and/or the aggregate number of itemsfor each brand, etc. Such information is a data product for which amarketplace may develop. (In some embodiments, items that are identicalin resin type, color, food/non-food, virgin/recycled, etc., may actuallybe sorted into different bins on the basis of their brand family, e.g.,if a particular brand wishes to use recyclate sourced from its ownplastic bottles.) Such concepts are further detailed in application Ser.No. 16/944,136.

In another variant embodiment, watermark information conveyed bycontainers is serialized, with each container bearing a differentidentifier. The processing facility can generate a log of whichserialized items are encountered in waste flows and processed forrecycling (or re-use). See, e.g., patent publications US20200193462 andWO2020136379. Containers bearing serialization information can bediverted into a bin for cleaning and re-use rather than processing intorecyclate.

In a further aspect, watermark-indicated data from the resolver databaseis used to train an AI system. In an exemplary arrangement a watermarkpayload is decoded from an item in a material stream, and the payload isresolved by a database lookup to obtain one or more metadata attributesabout the item (e.g., that the item is an Acme brand 500 ml waterbottle, made of PET plastic bearing a transparent polypropylene 4 milthick printed label, originally-capped with a green PVC cap, whichleaves a green PVC tamper-evident band (security ring) around the neckafter the cap is removed). One or more images of the bottle, captured bythe watermark camera system (or spectroscopy camera system, or AI camerasystem) are archived and labeled with the watermark-determined metadata.This process is repeated for some or all of the itemswatermark-identified from the material stream. These labeled imagesdepict items in various states of contamination and crushing, yet theirattributes are deterministically identified through use of watermarking.Over time a vast library of thousands (or millions) of suchaccurately-labeled images of items in material steams is accumulated.Such library can be used as training data for an image classifier (e.g.,a convolutional neural network as described in US20160063359 or U.S.Pat. No. 10,664,722), enabling the trained classifier to then provideprobabilistic estimates of such attribute metadata for an item depictedin a future image, based on imagery alone, without reliance onwatermarking. Initially the probabilistic estimates provided by such atrained classifier may be correct less than 90% of the time. But bytraining with increasing amounts of labeled imagery, the estimates gainin accuracy, perhaps reaching 90% or 95% or better, at least forsoil/crush presentations of items that are similar to those found in thetraining data.

To review:

An apparatus employing certain aspects of the technology comprises aconveyor, one or more cameras, and one or more light sources, to produceimagery of items on the conveyor. A first identification system applieswatermark decoding to the imagery to obtain first information about anitem on the conveyor. A second identification system appliesspectroscopy or neural network classification to said imagery to obtainsecond information about said item on the conveyor. A sorting logicprocessor is coupled to both the first and second identification systemsand configured to control one or more diverters in accordance withoutput data provided by said first and second identification systems.

In some embodiments the control unit of the apparatus is configured torespond to conflicting information provided by the first and secondidentification systems, by giving precedence to the first information ina first circumstance (e.g., when the item is black in color), and bygiving precedence to the second information in a second circumstance(e.g., when the item is not black in color).

Relatedly, another apparatus employing certain aspects of the technologycan comprise a first camera with associated light source that capturesimage data depicting items on a conveyor. The apparatus further includesa neural network classifier trained to identify a sub-region depicted inthe captured image data as belonging to one of plural classes, andconfigured to produce map data corresponding thereto. The apparatusstill further includes a second camera and associated light source, tocapture further image data depicting items on the conveyor. A codereader system (e.g., a watermark reader) is configured to analyze thefurther image data for coded data, and is responsive to the map datafrom the neural network classifier to limit its analysis to a sub-partof the further image data. The apparatus also includes a sorting logicprocessor coupled to at least the code reader system, configured tocontrol one or more diverters in accordance with output data provided bysaid code reader system.

As indicated, another waste sorting facility employing certain aspectsof the technology can comprise a near infrared imaging system includingone or more processors configured to discern a spectral signatureproduced by a plastic container, and to identify a plastic resin of thecontainer based on the spectral signature. The facility further includesa watermark imaging system including one or more processors configuredto extract an encoded watermark payload formed in a surface of theplastic container, or printed on a label of the plastic container, andto determine information from said watermark payload. The facilityfurther includes a processing system configured to make a sortingdecision for the plastic container based on the plastic resin identifiedby the near infrared imaging system, and based on the informationdetermined from the watermark payload.

In such arrangement the information determined from the watermarkpayload can include whether the container was used for food or non-food,and the system is configured to sort the container based on both theidentified plastic resin, and whether the container was used for food ornon-food.

In another such arrangement the information determined from thewatermark payload can include whether the container was formed of virginplastic or recycled plastic, and the system is configured to sort thecontainer based on both the identified plastic resin, and whether thecontainer was formed of virgin or recycled plastic.

Similarly, a further waste sorting facility employing certain aspects ofthe technology can include an artificial intelligence system (e.g.,comprising a convolutional neural network) that processes data,including image data, to make a judgment about an item in a wastestream. The facility also includes a watermark system including a cameraand one or more processors configured to extract an encoded watermarkpayload formed in a surface of the item, or printed on a label appliedto the item, and to determine information from said watermark payload.The facility further includes a diverter (e.g., a robotic arm) thatsorts the item from the waste stream based on the judgment made by theartificial intelligence system and based on the information determinedfrom said watermark payload.

A method employing aspects of the present technology can includecapturing a first image depicting a first item in a waste stream on aconveyor, and reading a first digital watermark payload encoded on thefirst item and depicted in the first image. A database is then accessedto determine, using the first digital watermark payload, that the firstitem is formed of polyethylene terephthalate, and was used to packagefood. Based on such information the first item is sorted into a firstcollection bin. The method further includes capturing a second imagedepicting a second item in the waste stream, and reading a seconddigital watermark payload encoded on the second item and depicted in thesecond image. The database is then accessed to determine, using thesecond digital watermark payload, that the second item is formed ofpolyethylene terephthalate, and was used to package non-food contents.Based on such information the second item is sorted into a secondcollection bin different than the first collection bin. Items in thefirst bin are sent for recovery of food-grade polyethylene terephthalaterecyclate, and items in the second bin are sent for recovery ofnon-food-grade polyethylene terephthalate recyclate.

A further apparatus employing certain aspects of the technology cancomprise a conveyor belt for moving a material stream of items past oneor more cameras that generate imagery. This imagery is input to firstand second identification systems that each produces one or moreattribute data about an item in said material stream. The firstidentification system comprises a watermark reading system. The secondidentification system comprises a spectroscopy identification system oran artificial intelligence identification system. The apparatus ischaracterized in that the one or more attribute data produced by thewatermark reading system includes food/non-food attribute dataindicating whether the item is a food container or a non-food container.The apparatus also includes a diverter system that directs items intodifferent repositories depending on a combination of plural attributedata. The plural attribute data includes attribute data provided by boththe first and second identification systems, including the food/non-foodattribute data produced by the watermark reading system.

A further method employing certain aspects of the technology can employfirst and second image processing systems that operate on imagerycaptured by one or more cameras viewing a waste stream on a conveyorbelt. The first system comprises a convolutional neural networkclassification system. The second system comprises a watermark detectionsystem. The method includes the convolutional neural networkclassification system classifying a first item on the conveyor belt andproviding data to the watermark detection system including locationinformation for the first item. The watermark detection system respondsto this data by not attempting a watermark reading operation on imagedata corresponding to said location information.

Warping

A tiled watermark signal can be warped prior to printing on a planarplastic sheet, in anticipation of the 3D shape that the sheet willfinally take. For example, warping can be applied to a tiled watermarksignal that is printed on a planar plastic sheet, which will later beshrunk-fit to a container, so that the finished diameter of theresulting sleeve will have a diameter that varies with height.

Consider the bottle profile depicted in FIG. 22 . It has three bulgesalong its height, each with one or two adjoining waists. A planar sheetis wrapped to form a cylindrical sleeve large enough to fit the bulges.Heat is then applied to shrink the sleeve, in places, to conform to thebottle shape.

This shrinking of the sleeve at the waists reduces the horizontal extentof any watermark blocks printed in these areas. To avoid differentialscaling of the watermark, applicant pre-warps the watermark blocks toreduce their vertical extent in such areas. By such arrangement, whenshrunk at the waist, the sleeve will present watermark blocks that areagain square. The blocks' side dimensions will be smaller than elsewhereon the bottle, but their lack of differential scaling simplifiesdecoding.

FIG. 23 shows this effect. The left side shows a uniform checkerboardpattern, shrunk-fit to a bottle waist. As can be seen, the horizontalshrinkage of the pattern at the waist leads to blocks that arevertically elongated. The right side shows applicant's technique. Bypre-warping the pattern to vertically-compress the blocks—in proportionto the bottle diameter—while the sleeve is still in its unshrunk state,the pattern after shrinking will present square blocks at the waist(albeit of smaller size than at the bulges).

A different problem arises if the watermark-printed substrate is not, atsome point, rectangular, yet wraps around a volume. Consider a plasticdrinking cup having a tapered shape. The diameter at the top is largerthan the diameter at the bottom. If unwrapped and laid flat, thesidewall has the shape of a sector of an annulus, e.g., aspartially-represented by FIG. 24A. If a pattern of square watermarkblocks spans such an annular sector, then at some point a troublesomepattern seam arises where the edges of the pattern meet. At this seamthe partial blocks do not transition smoothly—each to the next. Instead,the pattern abruptly stops at one boundary 481 (defined by a linethrough the pattern at a first angle), and meets a second boundary 482(defined by a line through the pattern at a second, different, angle). Atrouble with such a seam is that a watermark decoder—presented withimagery depicting such region of the cup—gets conflicting signals aboutthe orientation of the watermark. Is it oriented as suggested by thesignal on one side of the seam, or the other? Whichever decision ismade, the imagery on the opposite side of the seam contributes nothingto the decoding operation—except possible confusion. Decoding suffers.

In such instances, it is preferable to apply a polar warp to thewatermark signal blocks, as shown in FIG. 24B. Each square watermarkblock becomes a patch shaped as a sector of an annulus, with twostraight sides and two curved sides (the straight sides being oppositeeach other). This enables the edges 481 a, 482 a, to seamlesslytransition, provided that an integral number of watermark blocks areplaced around the circumference. Such polar warping avoids the decodingdifficulties of the FIG. 24A arrangement. The tradeoff is that the scaleof the watermark varies, e.g., from 160 to 193 WPI in an exemplary cup,with a 10 degree taper. However, existing watermark detectors cope wellwith such ranges of scale state variations, and they likewise have beenfound to cope well with polar distortions of this magnitude.

The mapping between locations in a repeating watermark block, andlocations in an annular sector, is detailed in the paper by Holubattached as an appendix to cited application U.S. 63/011,195.

Efficiently Handling Visual Code Transformations

Artisans understand that 2D codes on smooth surfaces can appear inverted(dark for light, etc.) when viewed in certain lighting conditions, anddecoding imagery of such codes can be attempted on both the original andinverted (re-inverted) forms of the imagery to address such possibility.(See, e.g., US patent documents 5,811,777, 20070295814 and 20090242649.)

Left-for-right mirroring is also a possibility, when a 2D code is formedon a first side of a plastic container and is sensed from the second,opposite side. This can occur if 3D texture marking of the first surfaceis strong enough to also deform the opposite surface. This can alsooccur if the container is transparent, and a marking formed on the firstside is viewed through the plastic from the opposite side.

A combination of inversion and mirroring can also arise.

Together with the normal presentation of a 2D mark on a surface, thereare thus four variants that may arise (normal, inverted, mirrored, andmirrored+inverted). Four attempts at decoding may thus be made, startingwith the original image. If no payload is recovered from the normalimage the image can be inverted, and a second attempt tried. If thatfails the original image can be mirrored and a third attempt tried. Ifthat fails the original image can be mirrored and inverted and a fourthattempt tried.

That is, a method employing aspects of the technology includesattempting a first time to locate a 2D code signal in captured image,failing the first time, and attempting a second time to locate aninverted code signal in the imagery. After failing in this attempt too,the method continues by attempting to locate a code signal that ismirrored, or both mirrored and inverted, in the imagery.

A naïve, brute force application of watermark decoding to the variouscases can be laborious. For example, following the method of our patentU.S. Pat. No. 9,959,587 to determine affine pose of a watermark withincaptured imagery requires that various operations, including 2D FFTs, beperformed four times for the four cases. However, much of thecomputational work performed for the first, normal, case can be adaptedand re-used for the other cases. This is because mathematical identitiesgenerally relate various of the computations involved in the differentcases.

For example, the watermark reference signal is a constellation of dozensof spatial frequency domain magnitude peaks of various phases. (Asnoted, FIG. 2B shows an illustrative magnitude peak constellation.)Applicant recognized the frequencies of these peaks are invariantthrough inversion and mirroring, so such frequency data does not need tobe computed four times. The frequencies of the reference signal peakscan be computed once, for the normal case, and the scale and rotation ofthe peaks' constellation reveals scale and rotation of the watermark forall four possible cases. This effects a substantial simplification.

After establishing scale and rotation of the watermark, the task ofestablishing (x,y) translation of the watermark block remains. Using thephase deviation approach detailed in the above-cited patent requiresestimating phases of each of the spatial frequency peaks, and thencalculating 1D phase deviation data, followed by calculating 2D phasedeviation data, and then iteratively refining.

The image in the left-for-right mirrored case is the same as the imagein the original case, except the x-coordinate of each pixel value isnegated. (For example, a pixel of value 103 found at pixel {72,176) inthe original image is now found at pixel (−72,176) in the mirroredimage, assuming the center pixel in the image block is given acoordinate of x=0.) This means that the phase θ_(N)(−u,v) of aparticular peak located at (−u,v) in the normal case becomes the phaseθ_(M) (u,v) at location (u,v) in the mirrored case (i.e., θ_(M)(u,v)=θ_(N) (−u,v)), it being understood that the (u,v) notation denotescoordinates in the Cartesian spatial frequency space within which thereference signal magnitude peaks are located.

The phase θ_(I)(u,v) of a particular peak (u,v) in the inverted case isthe phase of the same peak in the normal case, θ_(N)(u,v), plus πradians. That is θ_(I)(u,v)=θ_(N)(u,v)+π. (Care should be taken withwrapping to assure the phase remains between bounds of −π and π.)

By applying the negation of x coordinate to the spatial coordinates ofpixels in the mirrored case, and working through the math (e.g.,applying familiar identities such as sin(θ+π)=−sin θ), straightforwardrelationships can be likewise derived relating the peak phases in thenormal case to the peak phases for the mirrored, and mirrored+invertedcases.

By adopting such shortcuts, all four geometrical cases can be processedin about the time a naïve implementation handles the normal and invertedcases alone.

Furthermore, use of the above relations and the symmetry of thereference signal allows us to reuse the results of the 1D phasedeviation in the (non-inverted, non-mirrored) case with that in the(non-inverted, mirrored) case. Similarly, it allows us to reuse theresults of the 1D phase deviation in the (inverted, non-mirrored) casein the (inverted, mirrored) case.

Thus, a further aspect of applicant's technology involves analyzingimagery for multiple possible transformed presentations of a watermarkpattern, by analyzing one possible presentation of the watermarkpattern, and adapting intermediate results of that analysis to produceresults corresponding to one or more other of the possiblepresentations.

CONCLUDING REMARKS

It bears repeating that this specification builds on work detailed inthe earlier-cited patent filings, such as publications US20190306385 andWO2020186234. This application should be read as if those filings arebodily included here. (Their omission shortens the above text and thedrawings considerably, in compliance with guidance that patentapplications be concise.) Applicant intends, and hereby expresslyteaches, that the improvements detailed herein are to be applied in thecontext of the methods and arrangements detailed in the cited documents,and that such combinations form part of the teachings of the presentdisclosure.

While the focus of this disclosure has been on plastic containers, thetechnology is more broadly applicable. The detailed arrangements can beapplied to items formed of metal, glass, paper, cardboard and otherfibrous materials, etc. Similarly, while reference has often been madeto bottles, it will be recognized that the technology can be used inconjunction with any items, e.g., trays, tubs, pouches, cups, transportcontainers, etc.

Moreover, while the emphasis of the specification has been on recycling,it should be appreciated that the same technology can be used to sortitems for other purposes (e.g., for packages on a conveyor in awarehouse or shipping facility)

Although the described embodiments employ a reference signal comprisedof peaks in the Fourier magnitude domain, it should be recognized thatreference signals can exhibit fixed features in different transformdomains by which geometric synchronization can be achieved.

Relatedly, it is not necessary for a digital watermark signal to includea distinct reference signal for geometrical synchronization purposes.Sometimes the payload portion of the watermark signal, itself, has knownaspects or structure that enables geometrical synchronization withoutreliance on a separate reference signal.

The term “watermark” commonly denotes an indicia that escapes humanattention, i.e., is steganographic. While steganographic watermarks canbe advantageous, they are not essential. Watermarks forming overt,human-conspicuous patterns, can be employed in embodiments of thepresent technology.

For purposes of this patent application, a watermark is a 2D codeproduced through a process that represents a message of N symbols usingK output symbols, where the ratio N/K is less than 0.2. (Inconvolutional coding terms, this is the base rate, where smaller ratesindicate greater redundancy and thus greater robustness in conveyinginformation through noisy “channels”). In preferred embodiments theratio N/K is 0.1 or less. Due to the small base rate, a payload can bedecoded from a watermark even if half of more (commonly three-quartersor more) or the code is missing.

In a particular embodiment, 47 payload bits are concatenated with 24 CRCbits, and these 71 bits (“N”) are convolutionally encoded at a base rateof 1/13 to yield 924 bits (“K”). A further 100 bits of version data areappended to indicate version information, yielding the 1024 bitsreferenced earlier (which are then scrambled and spread to yield the16,384 values in a 128×128 continuous tone watermark).

Some other 2D codes make use of error correction, but not to such adegree. A QR code, for example, encoded with the highest possible errorcorrection level, can recover from only 30% loss of the code.

Preferred watermark embodiments are also characterized by asynchronization (reference) signal component that is expressed wheremessage data is also expressed. For example, every mark in a sparsewatermark is typically a function of the synchronization signal. Againin contrast, synchronization in QR codes is achieved by alignmentpatterns placed at three corners and at certain intermediate cells.Message data is expressed at none of these locations.

While a GTIN payload data field from a label watermark can be used toaccess attribute metadata (e.g., plastic type) from a database, this isnot required. Other fields of the label watermark can be used for thispurpose. Indeed, the use of a database in conjunction with labelwatermarks is not essential; the payload can convey plastic datadirectly, such as in one of the Application Identifier key value pairssupported by the standard governing GTINs (“GS1 General Specifications,Release 21.0.1, January 2021”).

Similarly, although GTIN information is commonly encoded in the labelwatermark only, in some embodiments the plastic texture watermark canencode this information as well. In such case, information about thecomponent plastic—or a destination sorting bin—can be obtained by use ofa data structure (such as a table) that associates the GTIN with suchother information.

In instances in which a shrink sleeve wraps a plastic bottle, the bottlesubstrate may be printed or textured to encode a first payload, whilethe sleeve may be printed or textured to encode a second payload. Thetwo payloads may be the same or different. If the same, the payload mayindicate the plastic composition of the underlying bottle, and mayadditionally indicate the plastic composition of the sleeve. Ifdifferent, the payloads may indicate the plastic composition of theplastic to which they are respectively applied.

Some recycling systems employ shredders to break down plastics intosmall pieces (e.g., on the order of 2, 1 or 0.5 cm across). In suchprocess a sleeve layer may separate from the substrate layer that itformerly wrapped. Shredding permits imaging of substrate surfaces thatwere formerly concealed, e.g., due to being adjacent the sleeve, orforming the interior of a bottle. From such imagery the encodedinformation (or parts thereof, such as a registration signal) can bedetected. Separation of pieces of different materials can be controlled(e.g., using air deflection systems) based on such information.

Other systems may sense the payload information encoded on sleevelayers, and route items having particular sleeve types to a strippingline for removal of such sleeves. Removal may there be accomplished bymechanical or chemical techniques. The underlying substrate can then beimaged, and routed or sorted as appropriate based on information decodedfrom its encoding.

While a RealSense 3D camera, based on stereovision principles, was citedabove, it will be understood that other 3D sensors, based on othertechnologies, can naturally be employed. These include structuredlight-based sensors, LIDAR and other time-of-flight sensors, etc.

Although the specification particularly details use of 2D and 3D imagesensors, 2D and 3D sensors are not required. Image sensing can insteadbe performed by a linear array sensor that captures line scan images ata suitably-high rate. (Some NIR spectroscopy systems employ such 1Dimage sensors.)

The noted Sony sensors, and others, have modes permitting image capturewithin only identified regions of interest (ROIs) within the field ofview. In applications in which the watermark reader knows it candisregard certain areas of the belt (e.g., based on information from anAI system, or a system that identifies vacant areas of the belt), suchROI feature can be used to capture pixel data over only a subset of thesensor field of view. Subsequent processing can then be applied just tothe ROI data provided by the sensor, improving efficiency.

Different ROIs can also be captured with different exposure intervalsconcurrently. Thus, if an AI system identifies both a dark object and alight object that will be within the watermark camera field of view,ROIs allocated by the watermark camera to the corresponding areas candiffer in exposure intervals, e.g., capturing data for 75 microsecondsin the darker area and 25 microseconds in the lighter area. The exposureintervals overlap in time, rather than being time-sequential. In stillother arrangements, two ROIs are defined over a common area within thefield of view and capture two sets of image data over two differentexposure intervals, e.g., 25 microseconds and 75 microseconds, whereagain the two different exposure intervals overlap in time. Depending onthe reflectance of the item within the common area, one of the twoexposures is likely to be either underexposed or overexposed. But theother of the two may depict the item with better watermark code contrastthan would be possible with a single intermediate exposure, e.g., of 50microseconds. The two exposures can be combined in known fashion toyield a high dynamic range image from which the watermark signal can beread.

Different exposures may also be captured in systems with lesssophisticated sensors, with similar opportunities and benefits. Forexample, a first frame can be captured with red light and a shortexposure, followed by a second frame captured with blue light and ashort exposure, followed by a third frame captured with red light and along exposure, followed by a fourth frame captured with blue light and along exposure, and then this cycle repeats. One of these frame capturesstarts every two milliseconds. (Long and short exposures are relative toeach other and can be, e.g., 75 and 25 microseconds.) Each capturedframe can be tagged with metadata indicating the illumination color andexposure interval, permitting the watermark detector to apply parametersoptimized to each circumstance.

In addition to gathering imagery for watermark decoding, spectroscopyidentification, neural network analysis, empty belt detection, etc., thecamera(s) noted above (or additional camera(s)) can detect bottles andother items that are rolling (tumbling) relative to the moving conveyorbelt. Uncrumpled bottles are susceptible to rolling in the circumstancesof the high belt speeds, induced winds, and generally chaotic dynamicsof waste stream conveyors, and such rolling interferes with accuratediversion of identified bottles by air-jets, robotic arms, etc. Byanalysis of imagery captured by a camera at two or more instants a knowninterval apart (or multiple cameras at two or more different instants),the speed and direction at which an item is tumbling—within the buildingframe of reference—can be determined.

The artisan will recognize that this is an exercise in photogrammetry,i.e., relating depicted positions of an item in image frames tocorresponding physical locations in the building by a projectionfunction specific to the camera system, and determining the time rate ofchange of such positions in two dimensions. If a bottle's speed therebyindicated is different than the belt speed, then the bottle is known tobe rolling. Given the known bottle rolling speed and direction, thediverter system can estimate the bottle's position at future instants,and can adapt the ejection timing or other parameters accordingly so thebottle is correctly diverted despite its rolling. Usually, the divertersystem will delay the moment of ejection, in accordance with thedifference between the bottle's speed and the belt speed.

That is, a method employing certain aspects of the technology includescapturing first imagery depicting waste on a conveyor, including aparticular item. The method further includes capturing second imagerydepicting the waste, including said particular item, on the conveyor.The captured imagery is processed to discern that the particular item ismoving at a different rate than said conveyor. As a consequence, adiverter is operated to remove the particular item from the waste on theconveyor, taking into account its moving at a different rate.

The belt speed can be detected by various means. One is to measure thetime interval with which a known mark on the top or bottom of the beltperiodically returns to a mark sensing station. Given such timeincrement, and the known length of the belt, the belt speed can becomputed. Alternatively, two images of the belt, captured by thewatermark reading camera, can be correlated to determine the pixeldistance traveled by the belt between the two image captures. This pixeldistance can be translated into a physical distance in the plane of thebelt by the camera system's projection function. Knowing this distance,and the interval between the two image captures, the belt speed againcan be computed.

Some embodiments are described as employing correlation as a method ofpattern matching (e.g., to determine vacant regions of belt). It will beunderstood that there are many variations of, and alternatives to,correlation, so the technology should be understood as encompassingother pattern matching techniques as well.

In certain of the embodiments, empty locations on the belt are detected,and processing resources that would normally be applied to detecting awatermark reference signal at such locations can be applied elsewhere.Naturally, such concept can be applied to other computationallyintensive tasks, such as recognizing items by artificial intelligencetechniques (e.g., convolutional neural networks, deep learning, etc.),by fingerprinting (e.g., SIFT and other feature point recognitionarrangements), optical character recognition, etc.

Reference was made to processing patches of captured imagery ofspecified sizes in waxels. While the exact waxel-size of a patch cannotbe determined until its scale is assessed (e.g., using the cited directleast squares method), the encoding scale of each watermark that thesystem might encounter is known in advance, and the imaging distance isfixed, so the scale-correspondence between captured pixels and encodedwaxels is roughly known, which is adequate for the present purposes.

As noted, captured imagery can be submitted to a convolutional neuralnetwork that has been trained to classify input imagery to identifydepicted object type. The object type can inform parameters of thediversion operation in addition to timing, such as the force to beapplied. For example, a flat object (e.g., a padded shipping envelope)can serve as a sail—capturing air, so less air is applied to divert aflat than is applied to divert a bottle (the curved surface of whichgenerally diverts the air around the bottle).

There is a short interval of time between the moment an item is imagedby the camera(s), and the moment the item is positioned for diversionfrom the conveyor. This interval may be sufficient to enable cloudprocessing. For example, captured imagery (or derivatives of suchimagery) can be transmitted to a remote cloud computer, etc. such asMicrosoft Azure, Google Cloud, Amazon AWS. The cloud processor(s) canperform some or all of the processing detailed herein, and return resultdata to the material processing system—in time to control the divertersaccordingly.

Likewise, in a material stream in which some items require a databaselookup to determine attribute metadata from an encoded containeridentifier, time may be adequate to permit a cloud database lookup priorto diversion.

Various references were made, above, to certain information encoded inthe watermark payload (e.g., identifying the plastic resin, the productbrand or the bottle manufacturer). It should be understood that suchinformation is often not literally encoded into the watermark payloaditself but is available from a database record that can be accessedusing an identifier that is literally encoded into the watermarkpayload. Applicant means language such as “information encoded in thewatermark” in this sense of “available from,” i.e., encompassing use ofa database to store the indicated information. (Applicant uses thephrase “literally encoded” to mean encoded in the stricter sense, i.e.,with certain information expressed by the watermark pattern on thebottle itself.)

This specification also frequently references “waste.” This is meant torefer simply to a material flow of used items. Some may be recycled;others may be re-used.

It will be recognized that recycling systems employing aspects of thepresent technology do not require a conveyor belt per se. For examples,articles can be transported past the camera system and to divertersystems otherwise, such as by rollers or by free-fall. All suchalternatives are intended to be included by the terms “conveyor belt,”“conveyor” or “belt.”

While reference was made to a few particular convolutional neuralnetwork architectures, it will be recognized that other CNNarchitectures suited for image classification can likewise be used.These include network arrangements known to artisans as AlexNet, VGG,Inception, ResNet, XCeption and DenseNet. Some image sensors includeintegrated neural network circuitry and can be trained to classifydifferent objects by their appearance, thus making such sensors suitablefor use in embodiments detailed above.

Although most of the detailed arrangements operate using greyscaleimagery, certain performance improvements (e.g., more reliableidentification of empty belt, and certain modes of watermark decoding)may be enabled by the greater-dimensionality of multi-channel imagery.RGB sensors can be used. However, half of the pixels in RGB sensors aretypically green-filtered (due to prevalence of the common Bayer colorfilter). Still better results can be achieved with sensors that outputfour (or more) different channels of data, such as R/G/B/ultraviolet. OrR/G/B/infrared. Or R/G/B/polarized. Or R/G/B/white.

Artisans will understand that the capture and distribution of imagery atthe high frame rates contemplated above is best performed by framegrabbers and other interface hardware adapted to such tasks. Exemplaryembodiments may include, e.g., the Kaya Predator frame grabber, and theMellanox Connect X5 Ethernet card. Such details are within the skill ofthe artisan so are not belabored here.

While the technology has been described in the context of digitalwatermarks, it will be recognized that any other machine-readablemarking can be used, such as DotCode and dot peen markings (althoughcertain benefits, such as readability from different viewpoints, may beimpaired). U.S. Pat. No. 8,727,220 teaches twenty different 2D codesthat can be embossed or molded into an outer surface of a plasticcontainer. If desired, an item may be marked with multiple instances ofa watermark or other 2D code block, with random noise interspersedbetween the blocks (e.g., as in publication US20110240739).

Although many consumer product companies may want texture markings to besubtle and easily overlooked, other may want such markings to beimmediately apparent and overt, e.g., to promote the fact that thecontainer was designed with recycling in mind.

While reference is often made to watermark blocks that are square inshape, it will be recognized that printed or textured surfaces canlikewise be tiled with watermark blocks of other shapes. For example, ahexagonal honeycomb shape may be composed of triangularly-shaped waxels.

Similarly, while repeated reference was made to watermark data encodedin a 128×128 waxel block, it will be recognized that such dimensions areexemplary. Larger or smaller blocks can naturally be used.

As reviewed above, watermark detection and synchronization in anexemplary embodiment employs a direct least squares (and phasedeviation) approach. Other techniques, however, can also be used. Oneexample is a coiled all-pose arrangement, as detailed in patentpublication US20190266749. Another option is to use an impulse matchedfilter approach, (e.g., correlating with a template comprised of peaks),as detailed in U.S. patent documents 10,242,434 and 6,590,996.

It will be recognized that processing a surface to effect a matte, orfrosted, finish is a form of 3D surface shaping/texturing, albeit on avery small scale. Generally, any non-inked treatment that changes asurface's bidirectional reflectance distribution function (BDRF) orsurface roughness is regarded as a 3D shaping/texturing operationherein.

While LED illumination is detailed, it is noted that some lightingapplications are transitioning to laser diodes (e.g., automotiveheadlamps). Laser diodes are similarly useful in embodiments of thepresent technology (e.g., with diffusor sheets or lenses), because theyoffer increased light output relative to LEDs, with consequentimprovements in exposure intervals, depth of field, etc.

Reference was made to forced air blowout as one means for diverting anitem from a material flow, such as from a conveyor belt. A particularair blowout arrangement is detailed patent publication US20190070618 andcomprises a linear array of solenoid-activated air jet nozzlespositioned below the very end of a conveyor belt, from which locationitems on the belt start free-falling under the forces of gravity andtheir own momentum. Without any air jet activity, items cascade off anddown from the end of the belt, and into a receptacle or onto anotherbelt positioned below. Items acted-on by one or more jets are divertedfrom this normal trajectory, and are diverted into a more remotereceptacle—typically by a jet oriented to have a horizontal componentaway from the belt, and a vertical component upwards. Other systems userobotic arms to pick items from a material stream and toss them intobins or onto other conveyors. These and other separation and sortingmechanisms are known to the artisan, e.g., from U.S. patent publicationsU.S. Pat. Nos. 5,209,355, 5,485,964, 5,615,778, 20040044436,20070158245, 20080257793, 20090152173, 20100282646, 20120031818,20120168354, 20170225199 and 20200338753. Operation of such diverters iscontrolled in accordance with the type of item identified, as detailedearlier.

The discussions involving sparse watermarks describe them as dark markson a lighter background, but this is not essential. In otherarrangements light marks on a darker background can be employed. In thecase of thresholded binary watermarks, for example, a continuous tonewatermark can be thresholded to identify the lightest elements of thewatermark, and spatially-corresponding white elements can be copied intoa dark signal block until a desired density of dots is achieved.Similarly, while applicant generally follows a practice in which smallersignal levels correspond to darker marks, the opposite practice cannaturally be used. More generally, the light/dark conventions observedin the detailed embodiments are not essential but are merely exemplary,with inverted arrangements being similarly possible, as will berecognized by the artisan.

In some embodiments imagery is locally inverted on a patchwork basis tocounteract specular reflection inversion prior to watermark decoding.Such work is detailed in application 63/156,866, filed Mar. 4, 2021.

From the foregoing examples it will be recognized that theearlier-detailed embodiments of our inventive work are exemplary only,and that the technology is not so limited.

Attention is particularly-drawn to cited application Ser. No.16/944,136. That application details work by a different team at thepresent assignee but dealing with the same recycling, etc., subjectmatter. That application details features, methods and arrangementswhich applicant intends be incorporated into embodiments of the presenttechnology. That application and this one should be read in concert toprovide a fuller understanding of the subject technology.

It will be understood that the methods and algorithms detailed above canbe executed using computer devices employing one or more processors, oneor more memories (e.g. RAM), storage (e.g., a disk or flash memory), auser interface (which may include, e.g., a keypad, a TFT LCD or OLEDdisplay screen, touch or other gesture sensors, together with softwareinstructions for providing a graphical user interface), interconnectionsbetween these elements (e.g., buses), and a wired or wireless interfacefor communicating with other devices.

The methods and algorithms detailed above can be implemented in avariety of different hardware processors, including a microprocessor, anASIC (Application Specific Integrated Circuit) and an FPGA (FieldProgrammable Gate Array). Hybrids of such arrangements can also beemployed.

By microprocessor, applicant means a particular structure, namely amultipurpose, clock-driven integrated circuit that includes both integerand floating point arithmetic logic units (ALUs), control logic, acollection of registers, and scratchpad memory (aka cache memory),linked by fixed bus interconnects. The control logic fetches instructioncodes from an external memory, and initiates a sequence of operationsrequired for the ALUs to carry out the instruction code. The instructioncodes are drawn from a limited vocabulary of instructions, which may beregarded as the microprocessor's native instruction set.

A particular implementation of one of the above-detailed processes on amicroprocessor—such as discerning affine pose parameters from awatermark reference signal in captured imagery, or decoding watermarkpayload data—involves first defining the sequence of algorithmoperations in a high level computer language, such as MatLab or C++(sometimes termed source code), and then using a commercially availablecompiler (such as the Intel C++ compiler) to generate machine code(i.e., instructions in the native instruction set, sometimes termedobject code) from the source code. (Both the source code and the machinecode are regarded as software instructions herein.) The process is thenexecuted by instructing the microprocessor to execute the compiled code.

Many microprocessors are now amalgamations of several simplermicroprocessors (termed “cores”). Such arrangement allows multipleoperations to be executed in parallel. (Some elements—such as the busstructure and cache memory may be shared between the cores.)

Examples of microprocessor structures include the Intel Xeon, Atom andCore-I series of devices, and various models from ARM and AMD. They areattractive choices in many applications because they are off-the-shelfcomponents. Implementation need not wait for custom design/fabrication.

Closely related to microprocessors are GPUs (Graphics Processing Units).GPUs are similar to microprocessors in that they include ALUs, controllogic, registers, cache, and fixed bus interconnects. However, thenative instruction sets of GPUs are commonly optimized for image/videoprocessing tasks, such as moving large blocks of data to and frommemory, and performing identical operations simultaneously on multiplesets of data. Other specialized tasks, such as rotating and translatingarrays of vertex data into different coordinate systems, andinterpolation, are also generally supported. The leading vendors of GPUhardware include Nvidia, ATI/AMD, and Intel. As used herein, Applicantintends references to microprocessors to also encompass GPUs.

GPUs are attractive structural choices for execution of certain of thedetailed algorithms, due to the nature of the data being processed, andthe opportunities for parallelism.

While microprocessors can be reprogrammed, by suitable software, toperform a variety of different algorithms, ASICs cannot. While aparticular Intel microprocessor might be programmed today to discernaffine pose parameters from a watermark reference signal, and programmedtomorrow to prepare a user's tax return, an ASIC structure does not havethis flexibility. Rather, an ASIC is designed and fabricated to serve adedicated task. It is purpose-built.

An ASIC structure comprises an array of circuitry that iscustom-designed to perform a particular function. There are two generalclasses: gate array (sometimes termed semi-custom), and full-custom. Inthe former, the hardware comprises a regular array of (typically)millions of digital logic gates (e.g., XOR and/or AND gates), fabricatedin diffusion layers and spread across a silicon substrate. Metallizationlayers, defining a custom interconnect, are then applied—permanentlylinking certain of the gates in a fixed topology. (A consequence of thishardware structure is that many of the fabricated gates—commonly amajority—are typically left unused.)

In full-custom ASICs, however, the arrangement of gates iscustom-designed to serve the intended purpose (e.g., to perform aspecified algorithm). The custom design makes more efficient use of theavailable substrate space—allowing shorter signal paths and higher speedperformance. Full-custom ASICs can also be fabricated to include analogcomponents, and other circuits.

Generally speaking, ASIC-based implementations of watermark detectorsand decoders offer higher performance, and consume less power, thanimplementations employing microprocessors. A drawback, however, is thesignificant time and expense required to design and fabricate circuitrythat is tailor-made for one particular application.

A particular implementation of any of the above-referenced processesusing an ASIC, e.g., for discerning affine pose parameters from awatermark reference signal in captured imagery, or decoding watermarkpayload data, again begins by defining the sequence of operations in asource code, such as MatLab or C++. However, instead of compiling to thenative instruction set of a multipurpose microprocessor, the source codeis compiled to a “hardware description language,” such as VHDL (an IEEEstandard), using a compiler such as HDLCoder (available from MathWorks).The VHDL output is then applied to a hardware synthesis program, such asDesign Compiler by Synopsis, HDL Designer by Mentor Graphics, orEncounter RTL Compiler by Cadence Design Systems. The hardware synthesisprogram provides output data specifying a particular array of electroniclogic gates that will realize the technology in hardware form, as aspecial-purpose machine dedicated to such purpose. This output data isthen provided to a semiconductor fabrication contractor, which uses itto produce the customized silicon part. (Suitable contractors includeTSMC, Global Foundries, and ON Semiconductors.)

A third hardware structure that can be used to execute theabove-detailed algorithms is an FPGA. An FPGA is a cousin to thesemi-custom gate array discussed above. However, instead of usingmetallization layers to define a fixed interconnect between a genericarray of gates, the interconnect is defined by a network of switchesthat can be electrically configured (and reconfigured) to be either onor off. The configuration data is stored in, and read from, an externalmemory. By such arrangement, the linking of the logic gates—and thus thefunctionality of the circuit—can be changed at will, by loadingdifferent configuration instructions from the memory, which reconfigurehow these interconnect switches are set.

FPGAs also differ from semi-custom gate arrays in that they commonly donot consist wholly of simple gates. Instead, FPGAs can include somelogic elements configured to perform complex combinational functions.Also, memory elements (e.g., flip-flops, but more typically completeblocks of RAM memory) can be included. Likewise with A/D and D/Aconverters. Again, the reconfigurable interconnect that characterizesFPGAs enables such additional elements to be incorporated at desiredlocations within a larger circuit.

Examples of FPGA structures include the Stratix FPGA from Intel, and theSpartan FPGA from Xilinx.

As with the other hardware structures, implementation of theabove-detailed processes on an FPGA begins by describing a process in ahigh level language. And, as with the ASIC implementation, the highlevel language is next compiled into VHDL. But then the interconnectconfiguration instructions are generated from the VHDL by a softwaretool specific to the family of FPGA being used (e.g., Stratix/Spartan).

Hybrids of the foregoing structures can also be used to perform thedetailed algorithms. One employs a microprocessor that is integrated ona substrate as a component of an ASIC. Such arrangement is termed aSystem on a Chip (SOC). Similarly, a microprocessor can be among theelements available for reconfigurable-interconnection with otherelements in an FPGA. Such arrangement may be termed a System on aProgrammable Chip (SORC).

Still another type of processor hardware is a neural network chip, e.g.,the Intel Nervana NNP-T, NNP-I and Loihi chips, the Google Edge TPUchip, and the Brainchip Akida neuromorphic SOC.

Software instructions for implementing the detailed functionality on theselected hardware can be authored by artisans without undueexperimentation from the descriptions provided herein, e.g., written inC, C++, Visual Basic, Java, Python, Tcl, Perl, Scheme, Ruby, Caffe,TensorFlow, etc., in conjunction with associated data.

Software and hardware configuration data/instructions are commonlystored as instructions in one or more data structures conveyed bytangible media, such as magnetic or optical discs, memory cards, ROM,etc., which may be accessed across a network. Some embodiments may beimplemented as embedded systems—special purpose computer systems inwhich operating system software and application software areindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

Different of the functionality can be implemented on different devices.Different tasks can be performed exclusively by one device or another,or execution can be distributed between devices. In like fashion,description of data being stored on a particular device is alsoexemplary; data can be stored anywhere: tc.al device, remote device, inthe cloud, distributed, etc.

Other recycling arrangements are taught in U.S. patent documents U.S.Pat. Nos. 4,644,151, 5,965,858, 6,390,368, 20060070928, 20140305851,20140365381, 20170225199, 20180056336, 20180065155, 20180349864, and20190030571. Alternate embodiments of the present technology employfeatures and arrangements from these cited documents.

This specification has discussed various embodiments. It should beunderstood that the methods, elements and concepts detailed inconnection with one embodiment can be combined with the methods,elements and concepts detailed in connection with other embodiments.While some such arrangements have been particularly described, many havenot—due to the number of permutations and combinations. Applicantsimilarly recognizes and intends that the methods, elements and conceptsof this specification can be combined, substituted and interchanged—notjust among and between themselves, but also with those known from thecited prior art. Moreover, it will be recognized that the detailedtechnology can be included with other technologies—current andupcoming—to advantageous effect. Implementation of such combinations isstraightforward to the artisan from the teachings provided in thisdisclosure.

While this disclosure has detailed particular ordering of acts andparticular combinations of elements, it will be recognized that othercontemplated methods may re-order acts (possibly omitting some andadding others), and other contemplated combinations may omit someelements and add others, etc.

Although disclosed as complete systems, sub-combinations of the detailedarrangements are also separately contemplated (e.g., omitting various ofthe features of a complete system).

While certain aspects of the technology have been described by referenceto illustrative methods, it will be recognized that apparatusesconfigured to perform the acts of such methods are also contemplated aspart of applicant's inventive work. Likewise, other aspects have beendescribed by reference to illustrative apparatus, and the methodologyperformed by such apparatus is likewise within the scope of the presenttechnology. Still further, tangible computer readable media containinginstructions for configuring a processor or other programmable system toperform such methods is also expressly contemplated.

To provide a comprehensive disclosure, while complying with the PatentAct's requirement of conciseness, applicant incorporates-by-referenceeach of the documents referenced herein. (Such materials areincorporated in their entireties, even if cited above in connection withspecific of their teachings.) These references disclose technologies andteachings that applicant intends be incorporated into the arrangementsdetailed herein, and into which the technologies and teachingspresently-detailed be incorporated.

In view of the wide variety of embodiments to which the principles andfeatures discussed above can be applied, it should be apparent that thedetailed embodiments are illustrative only, and should not be taken aslimiting the scope of the technology.

The invention claimed is:
 1. A waste sorting apparatus comprising: aconveyor; one or more cameras, and one or more light sources, to produceimagery of items on the conveyor; a first identification system thatapplies watermark decoding to said imagery to obtain first informationabout an item on the conveyor; a second identification system thatapplies spectroscopy or neural network classification to said imagery toobtain second information about said item on the conveyor; one or morediverters; and a sorting logic processor coupled to both the first andsecond identification systems, configured to control the one or morediverters in accordance with output data provided by said first andsecond identification systems; wherein the sorting logic processor isconfigured to respond to conflicting information provided by the firstand second identification systems, by giving precedence to the firstinformation in a first circumstance, and by giving precedence to thesecond information in a second circumstance.
 2. The apparatus of claim 1in which the second identification system is a neural network classifierconfigured to determine capping status of a bottle on the conveyor, thatis to classify a bottle as being capped or as being uncapped, whereinthe sorting logic processor is configured to control the one or morediverters in accordance with output data including said capping statusdetermined by said neural network classifier.
 3. The apparatus of claim1 in which the second identification system is a neural networkclassifier configured to determine contamination status of a containeron the conveyor, that is to classify a container as clean or not clean,wherein the sorting logic processor is configured to control the one ormore diverters in accordance with output data including saidcontamination status determined by said neural network classifier. 4.The apparatus of claim 1 in which the second identification system is aneural network classifier configured to determine state informationabout a container on the conveyor, wherein the sorting logic processoris configured to control the one or more diverters in accordance withoutput data including said state information determined by said neuralnetwork classifier.
 5. The apparatus of claim 1 in which the secondidentification system provides a confidence metric for the secondinformation, wherein the sorting logic processor is configured torespond to conflicting information provided by the first and secondidentification systems by giving precedence to the first informationwhen the confidence metric has a first state, and by giving precedenceto the second information when the confidence metric has a second state.6. The apparatus of claim 1 in which the sorting logic processor isconfigured to give precedence to the first information when the firstinformation indicates the item is a sleeved container comprising a firstsubstrate material wrapped by a second, different, sleeve material.
 7. Awaste sorting apparatus comprising: a near infrared imaging systemincluding one or more processors configured to discern a spectralsignature produced by a plastic container, and to identify a plasticresin of the container based on said spectral signature; a watermarkimaging system including one or more processors configured to extract anencoded watermark payload formed in a surface of said plastic container,or printed on a label of said plastic container, and to determineinformation from said watermark payload; and a processing systemconfigured to make a sorting decision for said plastic container basedon the plastic resin identified by the near infrared imaging system, andbased on the information determined from the watermark payload; whereinthe processing system is further configured to resolve a conflictbetween the plastic resin identified by the near infrared imagingsystem, and a plastic resin determined from said watermark payload. 8.An apparatus for sorting plastic items in a waste stream, comprising: acrusher that serves to reduce height variations of plastic surfaces inthe waste stream; an artificial intelligence system that processes data,including image data, to make a judgment about a plastic item in thewaste stream; a watermark system including a camera and one or moreprocessors configured to extract an encoded watermark payload formed asa texture in a surface of said item and to determine information fromsaid watermark payload; and a diverter that sorts said item from thewaste stream based on the judgment made by the artificial intelligencesystem and based on the information determined from said watermarkpayload.
 9. The waste sorting apparatus of claim 8 in which theartificial intelligence system comprises a convolutional neural networkthat processes said image data.
 10. The waste sorting apparatus of claim8 in which said diverter comprises a robotic arm.